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Abstract 


This  study  offers  a  comprehensive  set  of  approaches  for  procurement  officials 
to  structure  public  investment  decisions.  Designed  to  improve  acquisition  outcomes, 
the  “Economic  Evaluation  of  Alternatives”  (EEoA)  addresses  a  significant  weakness 
in  most  contemporary  military  applications  of  the  current  methodology — the  Analysis 
of  Alternatives  (AoAs).  While  AoAs  correctly  focus  on  lifecycle  costs  and  the 
operational  effectiveness  of  alternatives,  “affordability”  is  an  after-thought — at  best 
only  implicitly  addressed  as  a  weight  placed  on  cost  in  the  final  stages  of  the 
analysis.  In  sharp  contrast,  the  EEoA  encourages  senior  analysts  and  decision¬ 
makers  to  include  affordability  explicitly  and  up-front  in  structuring  an  AoA.  This 
requires  working  with  vendors  to  build  alternatives  based  on  a  reasonable  spectrum 
of  possible  funding  (budget  or  affordability)  scenarios.  A  key  difference  between 
traditional  AoAs  and  the  EEoA  approach  is  that  instead  of  modeling  competing 
vendors  as  points  in  cost-effectiveness  space,  the  EEoA  solicits  vendor  proposals  as 
functions  of  optimistic,  pessimistic,  and  most-likely  funding  (budget)  scenarios.  The 
Decision  Map  offered  in  the  concluding  section  of  this  study  provides  a 
comprehensive  guide  to  EEoA  for  practitioners.  This  study  also  illustrates  how,  by 
embedding  affordability  directly  into  an  AoA,  the  EEoA  approach  provides  a  unique 
opportunity  for  senior  leadership  to  achieve  a  significant  defense  acquisition 
reform — to  integrate  Requirements  Generation  and  Defense  Acquisition  with  the 
Planning,  Programming,  and  Budgeting  System  (PPBS),  reducing  future  costs  and 
improving  performance  and  schedules. 

Keywords:  Public  Investment,  Acquisition  Outcome,  Economic  Evaluation  of 
Alternatives  (EEoA),  Analysis  of  Alternatives  (AoAs),  Defense  Acquisition  Reform 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


Acknowledgments 


I  am  grateful  to  colleagues  at  the  Defense  Resources  Management  Institute 
(DRMI)  and  the  Graduate  School  of  Business  &  Public  Policy  (GSBPP)  at  the  Naval 
Postgraduate  School,  to  defense-economics  colleagues  at  the  RAND-organized 
sessions  of  the  Western  Economics  Association,  and  to  the  Honorable  Jacques 
Gansler  and  other  current  and  former  Pentagon  officials  for  ideas  and  inspiration 
that  led  to  this  study.  Given  the  lively  debates  that  surround  this  topic,  I  am  solely 
responsible  for  the  views  expressed  and  for  any  errors.  I  also  gratefully  acknowledge 
Admiral  Jim  Greene,  Dr.  Keith  Snider  and  GSBPP’s  Acquisition  Research  Program 
for  funding  this  study.  The  usual  disclaimers  apply. 


-  V  - 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


-  VI  - 


About  the  Author 


Dr.  Melese  joined  the  Naval  Postgraduate  School  in  1987  and  today  is 
Professor  of  Economics  at  the  Defense  Resources  Management  Institute  (DRMI). 

He  teaches  executive  management  courses  at  the  Naval  Postgraduate  School  and 
in  countries  around  the  world.  He  has  been  an  advisor  and  consultant  to  the  Joint 
Staff  and  the  Office  of  the  Secretary  of  Defense,  and  in  2008  helped  edit  the  DoD’s 
first  Strategic  Management  Plan.  He  has  published  extensively  on  a  variety  of  topics, 
including  a  recently  co-authored  paper  entitled  “SUCCESS:  A  New  Management 
Model  for  Government”  implemented  in  the  Joint  Staff.  At  the  request  of  NATO  HQ 
and  the  State  Department,  Dr.  Melese  has  represented  the  US  as  an  expert  in 
defense  management  and  public  budgeting  throughout  Europe.  He  recently 
organized  a  major  NATO  meeting  in  Monterey  on  the  occasion  of  the  60*^ 
Anniversary  of  the  alliance:  “Building  Integrity  and  Defense  Institution  Building.” 


Francois  Melese,  Ph.D. 

Professor  of  Economics 

Defense  Resources  Management  Institute  (DRMI) 

School  of  International  Graduate  Studies 

Naval  Postgraduate  School 

Monterey,  CA  93943-5000 

Tel:  831-229-3179 

E-mail:  fmelese@nps.edu 


VII  ■ 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


-  X  - 


Table  of  Contents 


I.  Introduction . 1 

A.  Making  the  Case  for  “Affordability” . 1 

I.  Integrating  Requirements  Generation,  Defense  Acquisition, 

and  PPBS . 5 

II.  Embedding  Affordability  Assessments  into  AoAs . 9 

III.  An  Intuitive  Guide  to  the  Economic  Evaluation  of 

Alternatives  (EEoA) . 11 

IV.  A  Critical  Evaluation  of  the  Status  Quo:  Two  Popular 

Decision  Criteria . 15 

A.  Bang-for-the-Buck  (Benefit/Cost  or  Effectiveness/Cost) 

Ratios . 18 

B.  Weighted  Averages  of  Cost  and  Effectiveness:  Assigning 

a  Weight  to  Cost . 22 

V.  Six  Ways  to  Structure  an  Economic  Evaluation  of 

Alternatives  (EEoA) . 31 

VI.  Conclusion . 47 

A.  A  Decision  Map  for  Decision-makers . 47 

List  of  References . 51 

Appendix  1.  Mathematical  Appendix . 59 

Initial  Distribution  List . 61 


-  xi  - 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


I. 


Introduction 


A.  Making  the  Case  for  “Affordability” 

“An  AoA  is  an  analytical  comparison  of  the  operational  effectiveness...  and 

life-cycle  cost  of  alternatives  that  satisfy  established  capability  needs.” 

(DoD,  2006,  July  7,  Section  3.3) 

America’s  security,  billions  of  taxpayer  dollars,  and  the  survival  of  US  soldiers 
can  all  hinge  on  an  Analysis  of  Alternatives  (AoA).^  Routinely  conducted  to  support 
major  military  investment  decisions,  the  AoA  is  a  core  component  of  the  defense 
acquisition  process.  AoAs  shape  future  forces,  influence  defense  spending,  and 
occasionally  redefine  the  defense  industry. 

This  paper  reveals  a  significant  weakness  in  the  Multiple-criteria  Decision¬ 
making  (MCDM)  approach  that  underpins  many  contemporary  AoAs.^The  weakness 
is  that  while  MCDM  techniques,  and  therefore  most  AoAs,  correctly  focus  on 
lifecycle  costs  and  the  operational  effectiveness  of  alternatives,  “affordability”  is  an 
after-thought,  often  only  implicitly  addressed  through  a  weight  assigned  to  costs.  As 
Norm  Augustine,  the  former  chairman  of  Lockheed  Martin  Corp.  (and  current  BENS 
task  force  chair  on  defense  acquisition  reform),  explains:  “affordability  is  rarely 
considered”  (as  cited  in  Newell,  2009). 

This  paper  offers  a  comprehensive  set  of  approaches  for  procurement 
officials  to  structure  public  investment  decisions  to  improve  acquisition  outcomes. 
Called  the  Economic  Evaluation  of  Alternatives  (EEoA),  these  approaches  address  a 
significant  weakness  in  most  contemporary  military  Analyses  of  Alternatives  (AoAs): 


^  This  study  often  uses  the  term  “Analysis  of  Alternatives”  (AoA)  in  its  broad,  generic  sense.  Although 
focused  on  defense  acquisition,  the  results  of  the  study  apply  to  any  public-sector  procurement.  It 
should  be  clear  in  context  whenever  the  term  AoA  references  Major  Defense  Acquisition  Programs 
(MDAPs)  or  the  acquisition  of  Major  Automated  Information  Systems  (MAISs). 

^  For  examples  of  the  MCDM  approach,  see  Clemen  (1996),  French  (1986),  Keeney  (1982;  1992), 
Keeney  &  Raiffa  (1976),  or  Kirkwood  (1997). 


affordability.  An  important  goal  of  the  EEoA  is  to  encourage  and  guide  analysts  and 
decision-makers  to  integrate  affordability  early  in  the  acquisition  process. 

In  sharp  contrast  to  traditional  AoAs,  the  Economic  Evaluation  of  Alternatives 
(EEoA)  explicitly  addresses  affordability  up-front.  This  requires  working  with  vendors 
to  build  proposals  based  on  funding  (resource/budget)  scenarios.  The  key  difference 
between  the  MCDM  approach  to  AoAs  and  the  EEoA  approach  is  that  instead  of 
modeling  decision  alternatives  from  competing  vendors  as  points  in  cost- 
effectiveness  space,  the  EEoA  generates  vendor  proposals  as  functions  of 
optimistic,  pessimistic,  and  most-likely  funding  (resource/budget)  scenarios. 

Given  the  current  economic  crisis  and  future  public-spending  challenges, 
affordability  is  a  growing  concern.  As  a  consequence,  it  is  imperative  that  the 
Department  of  Defense  (DoD)  obtain  the  best  value  for  every  dollar  it  invests  in  its 
Major  Defense  Acquisition  Programs  (MDAPs)  and  Major  Automated  Information 
Systems  (MAISs).  The  primary  goal  of  this  study  is  to  improve  defense  decisions  by 
bringing  the  taxpayer  up-front  alongside  the  warfighter  in  the  defense  acquisition 
process.  This  is  accomplished  by  explicitly  embedding  affordability  into  AoAs.  The 
US  Government  Accountability  Office  (GAO)  confirms  that  a  major  challenge  facing 
the  DoD  is  to  “achieve  a  balanced  mix  of  weapon  systems  that  are  affordable” 

(GAO,  2009,  March  18,  p.  5).^ 

Unlike  the  traditional  MCDM  approach  to  AoAs  that  focuses  on  costs  and 
operational  effectiveness  (schedule  and  performance),  the  EEoA  approach  adds 
another  dimension:  affordability.  The  EEoA  makes  a  clear  distinction  between  the 
“lifecycle  costs”  or  “price”  of  an  alternative,  its  operational  effectiveness  (schedule 
and  performance),  and  the  resources  (funding  or  budget)  likely  to  be  available  for 
the  overall  program.  By  embedding  affordability  directly  into  an  AoA,  the  EEoA 
approach  provides  a  unique  opportunity  for  senior  leadership  to  achieve  a  significant 


^  According  to  the  Government  Accountability  Office  (GAO),  over  the  next  5  years,  the  DoD  plans  to 
spend  more  than  $357  billion  on  development  and  procurement  of  major  defense  acquisition 
programs  (GAO,  2009,  March  18,  p.  4). 


defense  acquisition  reform — to  integrate  Requirements  Generation  and  Defense 
Acquisition  with  the  Planning,  Programming,  and  Budgeting  System  (PPBS), 
reducing  future  costs  and  improving  performance  and  schedules/ 


The  DoD’s  Planning,  Programming,  Budgeting  and  Execution  (PPBE)  process  is  the  principal 
decision-support  system  used  to  provide  the  best  possible  mix  offerees,  equipment,  and  support 
within  fiscal  constraints.  Two  other  major  decision  support  systems  complement  the  PPBE  process:  a 
Requirements  Generation  System  to  identify  military  investment  opportunities  and  the  Defense 
Acquisition  System  (DAS)  to  develop  and  procure  new  weapon  systems. 
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I.  Integrating  Requirements  Generation,  Defense 
Acquisition,  and  PPBS 


In  stressing  affordability,  the  Economic  Evaluation  of  Alternatives  (EEoA) 
offers  an  analytical  approach  that  addresses  a  major  concern  expressed  by  the  US 
Government  Accountability  Office  (GAO): 

DoD’s  processes  for  identifying  war-fighter  needs  [Requirements  Generation], 
allocating  resources  [Planning,  Programming,  and  Budgeting  System 
(PPBS)],  and  developing  and  procuring  weapon  systems  [Defense 
Acquisition]  are  fragmented  [so  that]  DoD  commits  to  more  programs  than 
resources  [budgets/funding]  can  support.  [...]  DoD  allows  programs  to  begin 
development  without  a  full  understanding  [of]  the  resources  [budgets/funding] 
needed.  (2009  p.1)= 

This  observation  is  echoed  in  a  recent  study  by  BENS  (Business  Executives 
for  National  Security):  “[R]equirements  are  largely  determined  [...]  without  adequate 
input  as  to  what  is  affordable  from  a  planning,  programming,  and  budgeting 
perspective"  (2009,  p.  10,  emphasis  added). 

By  embedding  affordability  directly  into  an  AoA  senior  leadership  has  a 
unique  opportunity  to  achieve  a  significant  defense  acquisition  reform:  to  integrate 
the  Requirements  Generation  System,  Defense  Acquisition  System  (DAS),  and  the 
Planning,  Programming,  and  Budgeting  System  (PPBS).  A  brief  overview  of  the 
DoD’s  budget  development  and  acquisition  systems  highlights  the  valuable  role 
affordability  can  play  in  structuring  an  EEoA. 


®  “The  lack  of  early  systems  engineering,  acceptance  of  unreliable  cost  estimates  based  on  overly 
optimistic  assumptions,  failure  to  commit  full  funding,  and  the  addition  of  new  requirements  well  into 
the  acquisition  cycle  all  contribute  to  poor  outcomes”  (GAO,  2009,  March  18,  p.1).  Whereas  this  study 
focuses  on  funding  risks,  Melese,  Franck,  Angelis  and  Dillard  (2007,  January)  introduce  an  economic 
approach  called  “Transaction  Cost  Analysis”  that  addresses  many  of  the  other  GAO  concerns. 
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The  primary  purpose  of  PPBS  is  to  make  hard  choices  among  alternative 
military  capabilities  and  investments  necessary  for  national  security  within  fiscal 
constraints.  The  requirements  generation  process  naturally  fits  into  the  Planning 
phase  of  PPBS.  The  first  step  in  any  investment  analysis  is  to  identify  the  derived 
demand  for  a  key  capability,  program,  or  project.  The  focus  of  the  EEoA  is  on 
materiel  investments  identified  to  fill  critical  capability  gaps.®  This  is  accomplished 
through  the  DoD’s  Requirements  Generation  System. 

Operator  demands  (“requirements”)  are  identified  and  refined  in  the  Planning 
phase  of  the  PPBE  process.  When  a  materiel  solution  is  recommended,  senior 
leaders  identify  prospective  military  investments  that  serve  as  the  basis  for  AoAs 
that  underpin  the  development  of  new  acquisition  programs  in  the  Defense 
Acquisition  System  (DAS).  Acquisition  proposals  (MDAP  and  MAIS)  that  emerge 
from  the  Planning  process  enter  the  DAS  and  are  incorporated  in  the  Programming 
phase  of  PPBE. 

Ideally,  the  Planning  phase  of  PPBE  establishes  fiscally  constrained  guidance 
and  priorities  for  the  military  services,  including  readiness,  sustainability  and 
modernization.  This  guidance  provides  direction  for  the  DoD  Components  (military 
departments  and  defense  agencies)  to  develop  their  individual  program  and 
investment  proposals,  or  Program  Objectives  Memorandum  (POM).  In  this 
Programming  phase  of  PPBE,  the  POMs  detail  resource-allocation  decisions 
(funding,  personnel,  etc.)  proposed  by  each  Component  for  its  programs,  projected 
six  years  into  the  future.  A  challenge  is  that  the  DAS  data  needed  to  evaluate  major 


®  Based  on  strategic-level  guidance  (the  National  Security  Strategy,  National  Military  Strategy, 
Quadrennial  Defense  Review,  Strategic  Planning  Guidance,  etc.),  the  requirements  generation 
system  reviews  existing  and  proposed  capabilities  and  identifies  critical  capability  gaps.  To  fill  those 
capability  gaps,  senior  leadership  examines  the  full  range  of  “doctrine,  organization,  training,  materiel, 
leadership  and  education,  personnel  and  facilities  (DOTMLPF)”  (CJCS,  2007,  p.  A-1;  USD  (AT&L), 
2008,  p.  14).  The  DAS  provides  principles  and  policies  that  govern  major  defense  acquisition 
decisions  and  milestones.  To  ensure  transparency  and  accountability,  and  to  promote  efficiency  and 
effectiveness,  various  instructions  (e.g.,  FAR,  DFARS,  DoD  Directive  5000.01,  DoD  Instruction 
5000.02,  etc.)  specify  statutory  and  regulatory  reports  (e.g.,  AoAs)  and  other  information 
requirements  for  each  milestone  and  decision  point. 


defense  acquisition  investments  in  an  EEoA  requires  lifecycle  cost  estimates  that 
project  well  beyond  the  six  years  of  the  POM. 

Senior  leadership  in  OSD  and  the  Joint  Staff  subsequently  review  each 
Component  POM  to  ensure  it  satisfies  the  Strategic  Planning  guidance  and  that  it 
can  be  integrated  into  an  effective  and  affordable  overall  national  defense  program. 
The  Budgeting  phase  of  PPBE  occurs  concurrently  with  the  Programming  phase. 

The  Budgeting  phase  converts  the  Programming  phase’s  (output-oriented) 
view  into  the  (input-oriented)  format  required  by  Congressional  appropriation 
structures.  While  the  DoD’s  biennial  defense  budget  projects  funding  only  two  years 
into  the  future,  it  includes  more  financial  detail  than  the  POMs.^  The  Under  Secretary 
of  Defense  Comptroller  and  the  Office  of  Management  and  Budget  (0MB)  jointly 
review  budget  submissions  to  ensure  programs  are  affordable,  i.e.,  satisfy  current 
fiscal  constraints.® 

The  primary  focus  of  Multi-criteria  Decision-making  (MCDM) — as  traditionally 
applied  in  AoAs — is  to  estimate  the  lifecycle  costs  and  operational  effectiveness  of 
alternative  defense  investments.  “An  AoA  is  an  analytical  comparison  of  the 
operational  effectiveness... and  Life-Cycle  Cost  of  alternatives  that  satisfy 
established  Capability  needs”  (DoD,  2006,  July  7,  Section  3.3).  This  paper 
emphasizes  another  key  aspect:  Affordability. 


^  By  translating  the  budget  implications  of  these  decisions  into  the  usual  Congressional  appropriation 
categories  (Military  Personnel,  Procurement,  Operations  &  Maintenance  (O&M),  Military  Construction, 
etc.),  decision-makers  are  able  to  generate  the  defense  budget  and  Future  Year  Defense  Program 
(FYDP). 

®  Office  of  Management  and  Budget  (0MB)  Circular  A-1 1,  titled  Preparation  and  Submission  of 
Budget  Estimates,  is  the  official  guidance  on  the  preparation  and  submission  of  budget  estimates  to 
Congress.  The  Army’s  Acquisition  guidance  emphasizes  “the  requirement  for  presenting  the  full 
funding  for  an  acquisition  program — that  is  the  total  cost  [for]  a  given  system  as  reflected  in  the  most 
recent  FYDP  [...]  pertains  to  all  acquisition  programs”  (DoA,  1999,  July  15,  p.  41). 
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In  helping  generate  investment  alternatives  and  illuminating  the  benefits  and 
costs  of  those  alternatives,  AoAs  have  the  potential  to  contribute  to  requirements 
generation  in  the  Planning  phase  of  PPBE  and,  through  DAS  decision  milestones, 
could  also  influence  the  Programming  phase  of  PPBE.  However,  according  to  the 
GAO:  “[T]he  vast  majority  of  capability  proposals  that  enter  the  JCIDS 
[Requirements  Generation]  process  are  [...]  approved  without  accounting  for 
resources  [funding/budgets]  that  will  be  needed  to  acquire  the  desired  capabilities” 
(GAO,  2009,  p.  6).®  The  concern  for  the  availability  of  resources  or  future  affordability 
is  a  primary  focus  of  this  paper. 

The  EEoA  approach  represents  an  important  step  in  the  long-running  effort  to 
integrate  the  DoD’s  Requirements  Generation  and  Defense  Acquisition  Systems 
with  PPBE.  In  considering  alternative  budget  scenarios  that  rely  on  FYDP  forecasts, 
policy-makers  can  use  the  EEoA  to  inject  an  explicit,  constrained  optimization 
approach  into  defense  acquisition  investment  decisions  that  parallels  one  already 
embedded  in  PPBE — choosing  an  optimal  mix  of  forces,  equipment  and  support  that 
maximizes  national  security  subject  to  fiscal  constraints. 


®  “A  2008  DoD  directive  established  nine  joint  capability-area  portfolios,  each  managed  by  civilian  and 
military  co-leads  [...].  However,  without  [...]  control  over  resources  [funding/budgets],  the  department 
is  at  risk  [...]  of  not  knowing  if  its  systems  are  being  developed  within  available  resources 
[funding/budgets]”  (GAO,  2009,  March  18,  p.  11). 
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II.  Embedding  Affordability  Assessments  into 
AoAs 

A  serious  concern  expressed  by  the  GAO  and  others  is  that  military 
requirements  are  often  approved  without  fully  accounting  for  resources  available, 
i.e.,  ignoring  future  funding  or  budget  realities.  This  reflects  a  fundamental  weakness 
in  the  way  AoAs  have  traditionally  been  structured.  While  decision-makers  use  AoAs 
to  estimate  the  costs  and  effectiveness  of  competing  alternatives,  affordability  is 
often  addressed  as  a  separate  exercise,  and  then  only  ex-post. This  is  reflected  in 
GAO’s  concern  that  “at  the  program  level,  the  key  cause  of  poor  outcomes  is  the 
approval  of  programs  with  business  cases  [AoAs]  that  contain  inadequate 
knowledge  about  [...]  resources  [i.e.  funding]  needed  to  execute  them”  (2009,  p.  7). 
Ironically,  this  directly  contradicts  the  defense  department’s  own  policy  outlined  in 
the  DoD  Directive  5000.01,  which  explicitly  states:  “All  participants  in  the  acquisition 
system  shall  recognize  the  reality  of  fiscal  constraints  [...].  DoD  components  shall 
plan  [. . .]  based  on  realistic  projections  of  the  dollars  [. . .]  likely  to  be  available  [and] 
the  user  shall  address  affordability  \r\  establishing  capability  needs”  (USD  (AT&L), 
2007,  p.  5,  emphasis  added). 

Current  DoD  directives  require  that  an  AoA  be  performed  at  key  milestone 
decision  points  (i.e..  A,  B,  C)  for  all  Major  Defense  Acquisition  Programs  (MDAP) 
and  Major  Acquisition  Information  Systems  (MAIS).  Affordability  assessments  (a 
separate  exercise)  are  only  required  at  Milestones  B  and  C  (USD  (AT&L),  2008, 
December  8,  p.  40).  The  Economic  Evaluation  of  Alternatives  (EEoA)  offers  a 
mechanism  to  embed  affordability  assessments  into  AoAs  as  early  as  Milestone  A. 
According  to  the  Defense  Acquisition  Guidebook,  the  purpose  of  an  affordability 


“Typically,  the  last  analytical  section  of  the  AoA  plan  deals  with  the  planned  approach  for  the  cost- 
effectiveness  comparisons  of  the  study  alternatives”  (DoD,  2006,  July  7,  Section  3.3).  Note  that  there 
is  no  mention  of  “affordability,”  but  instead  only  an  ex-post  cost-effectiveness  trade-off  that  implies  a 
concern  for  affordability.  Moreover,  this  trade-off  occurs  at  the  end  of  a  process  in  which  alternatives 
under  consideration  have  been  developed  independently  of  any  cost/budget/funding/affordability 
constraint.  The  US  Marine  Corps  (PA&E)  has  a  similar  approach  to  structuring  an  AoA. 
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assessment  is  to  demonstrate  that  a  program’s  projected  funding  requirements  are 
realistic  and  achievable."  “In  general,  the  assessment  should  address  program 
funding  over  the  six-year  programming  [FYDP]  period,  and  several  years  beyond. 
The  assessment  should  also  show  how  the  projected  funding  fits  within  the  overall 
DoD  Component  plan”  (DoD,  2006,  July  7,  Section  3.2.2).^^  In  preparing  affordability 
assessments,  one  possible  source  of  data  decision-makers  can  use  is  the  Future 
Years  Defense  Program  (FYDP).^^  The  EEoA  provides  a  mechanism  for  analysts 
and  decision-makers  to  embed  affordability  assessments  directly  into  AoAs. 


"  Since  this  assessment  requires  a  DoD  Component  corporate  perspective,  the  affordability 
assessment  should  not  be  prepared  by  the  program  manager,  nor  should  it  rely  too  heavily  on  the 
user.  It  requires  a  higher-level  perspective  capable  of  balancing  budget  trade-offs  (affordability) 
across  a  set  of  users  (DoD,  2006,  July  7,  Section  3.2.2). 

A  first  step  in  the  program’s  affordability  assessment  is  to  portray  the  projected  annual 
modernization  funding  (RDT&E  plus  procurement,  measured  as  TOA)  in  constant  dollars  for  the  six- 
year  programming  period  and  for  twelve  years  beyond.  Similar  funding  streams  for  other  acquisition 
programs  in  the  same  mission  area  also  would  be  included.  What  remains  to  be  determined  is 
whether  this  projected  funding  growth  is  realistically  affordable  relative  to  the  DoD  Component’s  most 
likely  overall  funding.  The  model  in  this  study  proposes  structuring  the  Economic  Evaluation  of 
Alternatives  not  only  for  a  “most  likely”  budget  but  also  for  an  “optimistic”  and  “pessimistic”  budget. 

An  output  of  the  DoD’s  PPBE  process,  the  Future  Year  Defense  Plan  (FYDP)  is  an  Office  of  the 
Secretary  of  Defense  (OSD)  database  that  contains  future  budget  projections. 
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III.  An  Intuitive  Guide  to  the  Economic  Evaluation 
of  Alternatives  (EEoA) 

In  order  to  nest  the  Requirements  Generation  and  Defense  Acquisition 
Systems  within  PPBE,  decision-makers  can  formulate  the  military’s  acquisition 
problem  in  terms  of  identifying  and  funding  investments  that  maximize  value 
(performance  or  effectiveness)  for  a  given  budget.  By  structuring  the  military 
investment  problem  as  a  constrained  optimization — i.e.,  maximizing  effectiveness 
subject  to  a  budget  constraint  (or  alternatively  minimizing  costs  of  obtaining  a  given 
level  of  effectiveness)^'' — decision-makers  would  boost  the  value  of  an  AoA  since  it 
now  directly  relies  on  higher-level  resource-allocation  decisions  in  the  Planning  and 
Programming  phases  of  PPBE. 

Unfortunately,  MCDM  techniques  typically  applied  to  structure  AoAs  do  not 
lend  themselves  to  this  interpretation.  As  a  consequence,  rather  than  being 
constrained  by  budgets,  budgets  are  typically  the  output  of  an  AoA,  generating  so- 
called  “funding  requirements.”'®  The  third  approach  to  structuring  an  EEoA  turns  this 
on  its  head.  Instead  of  generating  a  budget  through  the  AoA  process,  decision¬ 
makers  or  analysts  forecast  an  optimistic,  pessimistic,  and  most-likely  budget  as  part 
of  the  PPBE  process  and  then  challenge  vendors  to  propose  alternatives  that  fit 
within  that  budget  envelope.'®  This  offers  an  approach  to  Defense  investment 


These  dual-constrained-optimization  approaches  represent  the  first  two  of  six  ways  proposed  in 
this  study  to  structure  an  Economic  Evaluation  of  Alternatives  (EEoA). 

A  recent  Senate  Report  states  that  “Awards  are  made  on  the  basis  of  the  solicitation  of  factors  and 
sub-factors  by  a  Source  Selection  Official  who,  using  his  or  her  discretion  and  independent 
judgement  [e.g.,  guided  by  an  AoA],  makes  a  comparative  assessment  of  [...]  competing  proposals, 
trading  off  relative  benefits  and  costs”  (Chapter  1,  Commercial  Practices,  p.  65).  The  Senate 
Committee’s  recommendation  is  that  “Regulatory  guidance  [...]  be  provided  in  FAR  to  [include]  a 
minimum  weight  to  be  given  to  cost/price”  (p.  102).  Missing  in  this  discussion  is  an  explicit  and 
realistic  acknowledgement  of  “affordability” — the  resources,  funding,  or  budgets  available  for  the 
procurement — something  only  indirectly  and  implicitly  addressed  in  assigning  a  “weight”  to  cost  (see 
Section  2  of  this  study). 

'®  This  is  in  the  spirit  of  the  Department  of  the  Army’s  Acquisition  Procedures,  which  explicitly  states 
that  “Cost  as  an  Independent  Variable  (CAIV)  applies  to  all  defense  acquisition  programs  [...  and] 
treats  cost  as  an  input  to,  rather  than  an  output  of,  the  materiel  requirements  and  acquisition 
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decisions  based  on  explicit  funding  (resource/budget/affordability)  scenarios  that 
supports  the  “long-standing  DoD  policy  to  seek  full  funding  of  acquisition  programs” 
(DoD,  2006). 

According  to  the  Defense  Acquisition  Guidebook,  affordability  assessments 
should  also  provide  details  as  to  how  excess  funding  demands  will  be 
accommodated  by  reductions  in  other  mission  areas  or  in  other  accounts.^’'  This 
opportunity-cost  approach  is  an  illustration  of  the  last  of  six  ways  identified  in  this 
study  to  structure  an  EEoA. 

Whereas  funding  decisions  for  major  programs  take  place  through  the  PPBE 
process,  the  GAO  finds  that  “the  process  does  not  produce  an  accurate  picture  of 
the  department’s  resource  needs  [funding/budget  requirements]  for  weapon  system 
programs  [...].  Ultimately,  the  process  produces  more  demand  for  new  weapon 
system  programs  than  available  resources  can  support”^®  (Sullivan,  2009,  March  18, 

p.  6). 


The  EEoA  approach  directly  responds  to  these  challenges.  It  also  responds  to 
other  concerns  highlighted  by  the  GAO  that  continue  to  confront  the  DoD’s  Defense 
Acquisition  System,  including:  “(1)  [to  make]  better  decisions  about  which  programs 
should  be  pursued  or  not  pursued  given  existing  and  expected  funding;  [and]  (2)  [to 


processes”  (DoA,  1999,  p.  63).  The  Army  guidance  emphasizes  “CAIV  is  focused  on  [...]  meeting 
operational  requirements  with  a  solution  that  is  affordable  [...  and  that  does]  not  exceed  cost 
constraints  [and  to]  establish  CAIV-based  cost  objectives  (development,  procurement,  and 
sustainment  costs)  early  in  the  acquisition  process”  (p.  63).  Moreover,  the  “RFP  must  [...]  solicit  from 
potential  suppliers  an  approach  [...]  for  meeting  CAIV  objectives”  (p.  63). 

Note  that  in  the  off-year  of  the  biennial  PPBE  process,  the  DoD  Components  are  restricted  to  the 
second  year  of  the  biennial  budget  and  are  required  to  submit  Program  Change  Proposals  (PCPs) 
and/or  Budget  Change  Proposals  (BCPs)  to  account  for  any  program-cost  increases,  schedule 
delays,  etc.  PCPs  address  issues  over  a  multi-year  period,  whereas  BCPs  address  issues  focused  on 
the  upcoming  budget  year.  Moreover,  to  stay  within  fiscal  constraints,  BCPs  and  PCPs  must  identify 
resource  reductions  in  other  programs  to  offset  any  cost  growth.  This  is  similar  in  spirit  to  the 
opportunity-cost  approach  that  is  one  of  the  six  ways  proposed  herein  to  structure  an  EEoA. 

The  cost  of  many  programs  reviewed  by  the  GAO  exceeded  planned  funding/budget  levels  (GAO, 
2008,  July  2). 
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develop]  an  analytical  approach  to  better  prioritize  capability  needs”  (Sullivan  2009, 
March  18). 


In  generating  alternatives  under  optimistic,  pessimistic  and  most-likely  budget 
scenarios,  the  EEoA  requires  explicit  interaction  with  the  PPBE  process.  In  sharp 
contrast  with  the  MCDM  approach  that  underlies  most  AoAs,  the  EEoA  explicitly 
identifies  and  emphasizes  budget  and  funding  constraints  or  “affordability.” 
Widespread  adoption  of  the  EEoA  would  contribute  to  the  goal  of: 

greater  consultation  between  requirements,  budget,  and  acquisition 
processes  [that]  could  help  improve  the  department’s  [...]  portfolio  of  weapon 
programs  [...].  This  means  that  decision  makers  responsible  for  weapon 
system  requirements,  funding,  and  acquisition  execution  must  establish  an 
investment  strategy  in  concert  [...],  assuring  requirements  for  specific  weapon 
systems  are  clearly  defined  and  achievable  given  available  [funding/budget] 
resources.  (GAO,  2008,  July  2,  pp.  10,  14) 

The  next  section  offers  a  brief  description  and  critical  evaluation  of  the  status 
quo.  Two  common  decision  criteria  used  in  cost-effectiveness  analyses  are 
discussed.  The  first  is  the  popular  “bang-for-the-buck”  or  benefit/cost  ratio.  The 
second  criterion  involves  a  familiar  controversy  that  surrounds  the  choice  of  an 
appropriate  weight  to  assign  to  cost.  This  weighted-average-of-cost-and- 
effectiveness  decision  rule  originates  from  the  standard  static,  deterministic  MCDM 
approach  to  cost-effectiveness  analysis  that  underpins  most  contemporary  AoAs. 
Section  6  illustrates  six  alternative  recommended  approaches  to  structure  an 
Economic  Evaluation  of  Alternatives  (EEoA).^®  The  final  section  concludes  with  a 
Decision  Map  to  guide  analysts  and  decision-makers  in  selecting  which  of  the  six 
approaches  is  best  suited  to  their  circumstances. 


An  appendix  is  available  upon  request  that  reveals  the  static,  deterministic,  multi-stage, 
constrained-optimization,  micro-economic  production  (procurement  auction)  model  that  underpins  the 
central  EEoA  approach. 
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IV.  A  Critical  Evaluation  of  the  Status  Quo:  Two 
Popular  Decision  Criteria 

Today,  most  modern  military  investment  (and  disinvestment)  decisions  are 
supported  by  some  form  of  cost-benefit  analysis  (CBA).  The  US  DoD  applies  CBA  to 
anything  from  milestone  decisions  for  Major  Defense  Acquisition  Programs  (MDAPs 
and  MAISs),  to  outsourcing  (0MB,  2003,  May  29;  Eger  &  Wilsker,  2007),  to  public- 
private  partnerships,  privatization,  or  Base  Realignment  and  Closure  (BRAG)  actions 
(see  0MB  Circular  A-94;  FAR;  DFARS;  DoD  5000  series,  etc.). 

When  benefits  cannot  be  expressed  in  monetary  terms,  analysts  develop  so- 
called  “measures  of  effectiveness”  (MOEs),  in  which  case  CBA  is  generally  referred 
to  as  cost-effectiveness  analysis  (OMB,  1992,  October  29).^°  The  most  common 
methodology  and  approach  for  building  MOEs  and  structuring  cost-effectiveness 
analyses  is  alternately  referred  to  as  Multiple-criteria  Decision-making  (MCDM), 
Multi-attribute  Utility  Theory  (MAUT),  or  Multiple-objective  Decision-making  (MODM) 
(see  French  (1986);  Keeney  and  Raiffa  (1976);  Clemen  (1996);  Kirkwood  (1997); 
Parnell  (2006);  Ramesh  &  Zionts  (1997);  etc.). 

This  paper  describes  some  limitations  with  the  current  decision  criteria  used 
in  most  AoAs  and  proposes  an  alternate  methodology  derived  explicitly  from  the 
constrained-optimization  approach  recommended  for  an  EEoA.  The  latter  approach 
is  closer  in  spirit  to  the  economic  origins  of  cost-effectiveness  analysis  in  Gorman 
(1980),  Flitch  and  McKean  (1967),  Michael  and  Becker  (1973),  Stigler  (1945),  Theil 
(1952);  etc.,  but  is  most  often  attributed  to  Lancaster  (1966a;  1966b;  1971;  1979).  A 
key  difference  between  the  MCDM  approach  to  an  AoA  and  an  EEoA  is  that  instead 


Fisher  (1 965)  argues  that  “numerous  terms  [. . .]  convey  the  same  general  meaning  [. . .]  ‘cost-benefit 
analysis,’  ‘cost-effectiveness  analysis,’  ‘systems  analysis,’  ‘operations  analysis,’  etc.  Because  of  such 
terminological  confusion,  [...]  all  of  these  terms  are  rejected  and  ‘cost-utility  analysis’  is  employed 
instead”  (p.  185).  Although  this  study  uses  the  terms  “cost-benefit”  and  “cost-effectiveness” 
interchangeably,  the  assumption  throughout  is  that  neither  benefits  nor  effectiveness  can  be 
measured  in  monetary  terms. 
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of  modeling  decision  alternatives  from  competing  vendors  as  points  in  cost- 
effectiveness  space,  the  EEoA  models  alternative  vendor  proposals  as  functions  of 
optimistic,  pessimistic,  and  most-likely  funding  (resource/budget)  scenarios. 

The  EEoA  approach  directly  responds  to  the  GAO’s  observation  that 
affordability  needs  to  be  an  integral  part  of  any  business  case  (AoA):  “Our  body  of 
work  on  best  practices  has  shown  that  an  executable  business  case  is  one  that 
provides  demonstrated  evidence  that... the  chosen  concept  can  be  developed  and 
produced  within  existing  resources — that  is... funding...”  (GAO,  2010,  Jan  20,  p.  2). 
Benchmarking  against  the  private  sector,  the  GAO  emphasizes  that  “successful 
commercial  enterprises  [...]  follow  a  disciplined  integrated  process  during  which  the 
pros  and  cons  of  competing  proposals  are  assessed  based  on  strategic  objectives 
[...]  and  available  [budget/funding]  resources”  (GAO,  2009,  March  18,  p.  5). 

A  distinctive  feature  of  defense  investment  decisions  is  that  multiple  criteria 
such  as  cost  and  effectiveness  cannot  easily  be  combined  into  a  single,  overall 
objective  such  as  “government  profitability.”  The  problem  of  ranking  public 
investments  when  benefits  cannot  be  expressed  in  dollars  has  spawned  an 
extensive  literature  in  management  science,  operations  research  and  the  decision 
sciences. 

This  literature  models  investment  alternatives  as  bundles  of  measurable 
characteristics  (attributes  or  criteria).  Techniques  that  mostly  fall  under  the  umbrella 
of  MCDM  are  routinely  used  by  analysts  and  decision-makers  (e.g.,  through  AoAs) 
to  guide  public  investment  decisions.  The  development  of  Measures  of  Effectiveness 
(MOEs)^^  and  lifecycle  cost  calculations  are  used  to  help  rank  alternatives.  An 
ongoing  concern  is  how  to  integrate  costs  and  effectiveness  in  the  final  selection 


The  Defense  Acquisition  Guidebook  Section  3.3.1:  AoA  Plan  states  that  “measures  of  effectiveness 
[...]  provide  the  details  that  allow  the  proficiency  of  each  alternative  in  performing  the  mission  tasks  to 
be  quantified  [...].  A  measure  of  performance  typically  is  a  quantitative  measure  of  a  system 
characteristic  (e.g.,  range,  [...]  logistics  footprint,  etc.)  chosen  to  enable  calculation  of  one  or  more 
measures  of  effectiveness”  (DoD,  2006,  July  7). 
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process  (see  Henry  and  Hogan  (1995);  Melese  and  Bonsper  (1996,  December); 
Melese,  Stroup,  and  Lowe  (1997);  etc.)- 

In  their  pioneering  work  applying  economic  analysis  to  defense.  Hitch  and 
McKean  (1967)  define  a  “criterion”  as  the  “test  by  which  we  choose  one  alternative 
[...]  rather  than  another”  (p.  120).  They  stress  that  “[t]he  choice  of  an  appropriate 
economic  criterion  is  [...]  the  central  problem  in  designing  a  [cost-effectiveness] 
analysis”  (p.  158). 

The  two  most  popular  decision  criteria  used  to  integrate  cost  and 
effectiveness  in  AoAs  are  1)  to  construct  Benefit/Cost  (or  MOE/Cost)  ratios,  and  2) 
to  assign  a  weight  on  cost  relative  to  effectiveness  and  construct  a  weighted 
average  of  cost  and  effectiveness  (using  a  linear,  separable,  additive  “value” 
function).  The  latter  decision  criterion  is  a  common  prescription  for  AoAs  that 
emerges  from  MCDM.  Both  approaches,  however,  are  problematic.  We  first  focus  on 
what  is  arguably  the  most  commonly  applied  criterion:  benefit/cost  ratios.  Next,  we 
move  to  the  most  common  MCDM  decision  criterion:  assigning  a  relative  weight  to 
the  cost  (price)  of  alternatives  in  an  overall  value  function.  At  first  glance,  the 
Benefit/Cost  (MOE/Cost)  ratio  or  “bang-for-the-buck”  criterion  is  appealing. 

However,  it  turns  out  to  be  largely  meaningless  unless  alternatives  are  constructed 
for  a  specific  budget  (funding/affordability)  scenario  or  to  achieve  a  specific  level  of 
effectiveness.  Meanwhile,  the  second  decision  criterion  also  turns  out  to  be 
misleading  in  the  absence  of  a  specific  budget  (funding/affordability)  scenario  and  a 
good  understanding  of  opportunity  costs.^^ 


Ironically,  if  a  budget  scenario  is  specified,  there  is  no  need  to  take  the  MCDM  approach  that 
underpins  most  AoAs,  since  it  is  possible  to  adopt  the  EEoA  approach.  The  EEoA  approach 
constructs  alternatives  to  fit  within  a  budget  envelope,  converting  the  problem  into  a  straightforward 
MOE  maximization  (see  Section  6). 
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A.  Bang-for-the-Buck  (Benefit/Cost  or  Effectiveness/Cost) 
Ratios 

It  is  well  known  that  the  application  of  a  benefit/cost  ratio  (or  bang-for-the- 
buck)  decision  criterion  to  rank  alternatives  is  largely  meaningless  unless 
alternatives  are  constructed  for  a  specific  budget  (funding)  scenario,  or  to  achieve  a 
specific  level  of  effectiveness.  Yet,  the  next  three  examples  illustrate  that  this 
continues  to  remain  a  popular  decision  criterion  in  the  absence  of  budget  or 
effectiveness  constraints. 

1 .  In  a  military  text  entitled  Executive  Decision  Making,  the  author  offers 
that  “[w]hen  we  cannot  fix  cost  or  effectiveness,  we  might  combine 
them  to  help  us  choose  between  alternatives  [...].  If  neither  can  be 
fixed  [...]  we  can  establish  a  cost/effectiveness  ratio”  (Murray,  2002,  p. 
6-3,  6-10). 

2.  In  presenting  what  they  claim  is  a  “novel  cost-benefit  analysis”  for  the 
“comprehensive  evaluation  of  competing  military  systems,”  Byrns, 
Corban  and  Ingalls  (1995)  in  their  article  in  the  Acquisition  Review 
Quarterly  deTme  a  “merit  function”  as  “a  single  number  [...]  [that] 
reflects  the  ratio  of  benefits  derived  to  dollars  spent”  (i.e.,  a 
benefit/cost  ratio).  They  assert;  “with  this  [...]  approach,  the  cost 
effectiveness  of  competing  systems  can  be  compared  and  “provides 
for  objective  and  reliable  decision  making,”  where  “a  large  system 
merit  [benefit/cost  ratio]  is  preferable  to  a  small  one”  (p.2&4). 
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3.  Similarly,  in  a  section  entitled  Comparing  Costs  and  Benefits,  the 
Department  of  the  Army’s  Economic  Analysis  Manual  states:  “When 
the  results  yield  unequal  cost  and  unequal  benefits  [...],  all  alternatives 
[...]  may  be  ranked  in  decreasing  order  of  their  benefit/cost  ratios” 
(DoA,  2001,  February,  p.  32).^^ 

Each  of  these  examples  recommends  using  a  benefit/cost  ratio  as  the  decision 
criterion.  Each  also  neglects  including  either  an  affordability  constraint  (keeping  the 
level  of  cost  fixed),  or  a  performance  constraint  (keeping  the  level  of 
effectiveness  fixed).  Unfortunately,  a  noted  RAND  analyst,  the  legendary  Gene 
Fisher  (1971),  clearly  points  out  in  his  classic  text  Cost  Considerations  in  Systems 
Analysis: 

The  use  of  [benefit/cost]  ratios  usually  poses  no  problem  as  long  as  the 
analysis  is  conducted  in  [a]  framework  [...]  with  the  level  of  effectiveness  or 
cost  fixed.  However,  it  is  common  to  encounter  studies  where  this  has  not 
been  done,  with  the  result  that  the  comparisons  [are]  essentially  meaningless. 

(p.  11) 


A  fourth  example  involves  a  recent  landmark  RAND  study  on  Capabilities-based  Planning.  The 
author  falls  into  the  same  trap.  In  a  section  entitled  Choosing  Among  Options  in  a  Portfoiio,  Paul 
Davis  (2002)  develops  “A  Notional  Scorecard  for  Assessing  Alternatives  in  a  Portfolio  Framework,” 
where  alternatives  differ  in  both  their  costs  and  effectiveness.  Nevertheless,  the  decision  criterion 
recommended  by  the  author  to  select  an  alternative  in  “[t]he  last  column  is  the  ratio  of  effectiveness 
over  cost”  (p.  45-46). 
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Is  A1  really  superior  to  A2  ? 

LESSON:  DANGER  in  using  Benefit/Cost  (Bang/Buck)  or 
Cost/Benefit  (Buck/Bang)  ratios  without  anchoring  Budget  or  MOE 

▲  RANKING: 
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“The  perceived  benefits  of  the 
higher  priced  proposal  shall 
merit  the  additional  cost...” 

FAR  15.101-1(2)c 


Cost 


Figure  1.  Inappropriate  Application  of  Benefit/Cost  Ratios 

A  simple,  extreme  example  helps  illustrate  the  danger  in  using  benefit/cost 
ratios  without  anchoring  either  the  budget  or  a  specified  Measure  of  Effectiveness 
(MOE).  Suppose  Alternative  A1  in  Figure  1  costs  $10  million  and  yields  an  MOE  of 
10  utils  while  Alternative  A2  costs  $1  billion  and  yields  an  MOE  of  900  utils. 

Applying  a  benefit/cost  ratio  criterion  indicates  that  A1  has  a  bigger  bang-for-the- 
buck  since  it  returns  1  util  per  million  dollars,  while  A2  only  offers  0.9  utils  per  million 
dollars. 


However,  using  benefit/cost  ratios  to  rank  alternatives  is  dangerous  in  this 
case  because  it  ignores  the  absolute  magnitude  of  the  costs  involved.  Suppose  the 
situation  was  reversed,  and  A2  offered  a  higher  benefit/cost  ratio  than  A1 .  Anyone 
that  ignores  affordability  and  chooses  A2  strictly  on  the  basis  of  bang-for-the-buck 
would  be  in  for  an  unpleasant  surprise  (a  $1  billion  versus  a  $10  million  decision). 

Since  affordability  and  opportunity  costs  are  always  a  concern  in  public 
investment  decisions  (especially  through  the  Requirements  Generation  System, 
Defense  Acquisition  System,  and  PPBS),  it  is  imperative  that  analysts  and  decision- 


In  Figure  1,  the  slope  of  any  ray  from  the  origin  represents  a  constant  benefit/cost  ratio  anywhere 
along  that  ray.  The  steeper  the  slope,  the  greater  the  benefit/cost  ratio. 
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makers  explore  budget  and  opportunity  cost  implications  of  going  with  the  high-cost 
alternative  (for  example,  the  extra  $990  million  to  obtain  an  additional  890  utils  of 
MOE)  or,  equivalently,  of  the  savings  in  going  with  the  low-cost  alternative. 

In  applying  economic  analysis  to  defense.  Hitch  and  McKean  (1967)  warn: 

One  common  “compromise  criteria”  is  to  pick  that  [alternative]  which  has  the 
highest  ratio  of  effectiveness  to  cost.  [Mjaximizing  this  ratio  is  the  [decision] 
criterion.  [While]  it  may  be  a  plausible  criterion  at  first  glance  [...]  it  allows  the 
absolute  magnitude  of  [effectiveness]  or  cost  to  roam  at  will.  In  fact,  the  only 
way  to  know  what  such  a  ratio  really  means  is  to  tighten  the  constraint  until 
either  a  single  budget  (or  particular  degree  of  effectiveness)  is  specified.  And 
at  that  juncture,  the  ratio  reduces  itself  to  the  test  of  maximum  effectiveness 
for  a  given  budget  (or  a  specified  effectiveness  at  minimum  cost),  and  might 
better  have  been  put  that  way  at  the  outset,  (pp.  165-167)^® 

The  Economic  Evaluation  of  Alternatives  (EEoA)  approach  follows  this  advice  and 
other  key  recommendations  made  by  Hitch  and  McKean  (1967): 

The  test  of  maximum  effectiveness  for  a  given  budget  (or  alternatively, 
minimum  cost  of  achieving  a  specified  level  of  effectiveness)  [...]  seems 
much  less  likely  to  mislead  the  unwary.  (p.167) 

As  a  starter  [...]  several  budget  sizes  can  be  assumed.  If  the  same 
[alternative]  is  preferred  for  all  [...]  budgets,  that  system  is  dominant  [...].  If 
the  same  [alternative]  is  not  dominant  the  use  of  several  [...]  budgets  is 
nevertheless  an  essential  step,  because  it  provides  vital  information  to  the 
decision  maker,  (p.  176) 

The  conclusion  is  straightforward.  The  use  of  benefit/cost  ratios  as  a  decision 
criterion  in  AoAs  does  not  pose  a  problem  as  long  as  the  analysis  is  structured  in  a 
way  that  pays  close  attention  to  affordability  (i.e.,  for  a  fixed  budget 
(funding/affordability  level)),  or  performance  (i.e.,  for  a  fixed  level  of  effectiveness 
(MOE)).^®  Since  AoAs  typically  consider  alternatives  that  differ  in  both  their  costs 


The  authors  continue:  “Of  course,  if  the  ratios  did  not  alter  with  changes  in  the  scale  of 
achievement  (or  cost,  the  higher  ratio  would  indicate  the  preferred  system,  no  matter  what  the  scale 
[. . .].  But  to  assume  that  such  ratios  are  constant  is  inadmissible  some  of  the  time  and  hazardous  the 
rest”  (Hitch  &  McKean,  1967,  p.  167). 

^®  An  additional  (necessary  and  sufficient)  condition  is  a  linear,  separable,  additive  objective  function. 
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(price)  and  benefits  (MOEs),  the  use  of  benefit/cost  (or  effectiveness/cost)  ratios  to 
rank  alternatives  is  at  best  “misleading. Partly  as  a  consequence,  decision 
scientists  developed  another  decision  criterion — Multi-criteria  Decision-making 
(MCDM) — to  rank  investment  options;  MCDM  is  routinely  applied  in  AoAs.  This 
second  popular  decision  criterion  is  examined  below. 

B.  Weighted  Averages  of  Cost  and  Effectiveness:  Assigning 
a  Weight  to  Cost 

MCDM  is  often  used  as  an  umbrella  term,  and  that  is  how  it  will  be  used  in 
this  study.  “In  the  literature  the  terms  multi-attribute  decision  making  (MADM),  multi¬ 
criteria  decision  making  (MCDM),  and  multi-objective  decision  making  (MODM)  are 
used  almost  interchangeably”  (French,  1986,  p.  105).  In  a  typical  MCDM  evaluation, 
a  decision-maker  (DM)  is  asked  to  identify  desired  attributes  (criteria/characteristics) 
of  a  project,  program  or  system  to  fill  some  critical  capability  gap  given  a  specific 
threat  scenario.  Next,  the  DM  is  asked  to  reveal  agreeable  trade-offs  among  those 
attributes.  An  exercise  of  this  sort  helps  analysts  uncover  the  DM’s  underlying  trade¬ 
offs,  or  their  “utility”  function,  to  generate  a  Measure  of  Effectiveness  (MCE)  for  each 
alternative.^® 

To  uncover  a  DM’s  utility  function,  decision  scientists  beginning  with  Saaty 
(1977)  bridged  an  important  implementation  gap.  Multiple  objective  (analytic) 
hierarchy  approaches  were  developed  to  help  reveal  underlying  utility  functions.  For 
example,  an  objectives  hierarchy  can  help  a  DM  work  down  from  a  high-level 
objective  (provide  national  security)  to  a  relevant  set  of  sub-objectives  (an  effective 


“Usually,  ratios  are  regarded  as  potentially  misleading  because  they  mask  important  information” 
(DoD,  2006,  July  7,  Section  3.3.1). 

“Measures  of  Effectiveness  [...]  provide  the  details  that  allow  the  proficiency  of  each  alternative  in 
performing  the  mission  tasks  to  be  quantified  [...].  A  measure  of  performance  typically  is  a 
quantitative  measure  of  a  system  characteristic  (e.g.,  range,  etc.)  chosen  to  enable  calculation  of  one 
or  more  measures  of  effectiveness  [...].  The  cost  analysis  normally  is  performed  in  parallel  with  the 
operational  effectiveness  analysis.  It  is  equal  in  importance  in  the  overall  AoA  process  [...].  [I]ts 
results  are  later  combined  with  the  operational  effectiveness  analysis  to  portray  cost-effectiveness 
comparisons”  (DoD,  2006,  July  7,  Section  3.3.1) 
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airlift  capability),  to  specific  attributes  (mobility,  transportability,  etc.)  and,  finally,  to 
measurable  characteristics  (Mobility  =  speed  (S),  range  (R);  transportability  = 
payload  (P),  weight  (W),  etc.).  The  outcome  in  this  example  is  a  utility  function  for 
airlift  capability:  U=U(M(S,R);T(P,W)),  where  the  characteristics  might  be  measured 
respectively  in  mph,  miles,  cubic  feet,  and  pounds. 

The  standard  assumption  in  the  literature  is  to  define  a  linear,  separable, 
additive  utility  function  that  generates  an  MOE  for  each  alternative — roughly 
analogous  to  a  weighted  average  of  its  attributes — provided  certain  assumptions  are 
satisfied,  such  as  “preferential  independence,”  etc.  (see  French  (1986);  Keeney  and 
Raiffa  (1976);  and  Keeney  (1994)).  There  is  a  vast  literature  concerned  with  eliciting 
preference  weights  and  the  normalization  of  characteristics  data  that  involves 
several  important  issues  reserved  for  future  research.^® 

Temporarily  overlooking  these  issues,  it  is  interesting  to  note  in  passing  that 
maximizing  a  linear  multi-attribute  utility  function  subject  to  a  budget  (affordability) 
constraint  yields  a  decision  rule  analogous  to  the  benefit/cost  ratio  criterion 
discussed  above.  Under  the  assumption  of  a  fixed  budget  and  linear,  additive, 
separable  utility  function,  the  benefit/cost  decision  rule  can  be  used  to  rank 
alternatives.  In  this  case,  the  winning  alternative  is  the  one  that  generates  the 
highest  MOE  per  dollar  or  the  biggest  bang-for-the-buck.  With  a  more  general  (non¬ 
linear)  utility  function,  the  equivalent  optimization  generates  a  more  complex, 
marginal  benefit/marginal  cost  decision  rule. 

In  reality,  MCDM  techniques  that  underpin  the  configuration  of  the 
fundamental  decision  problem  in  most  AoAs  do  not  rely  on  an  explicit  discussion  of 


For  example,  one  issue  is  that  normalization  of  attributes  is  not  necessary,  and  worse,  can  be 
misleading.  The  author  is  aware  of  several  applications  in  which  relative  weights  were  assigned  to 
different  attributes  based  on  soliciting  acceptable  trade-offs  among  measurable  characteristics  from 
decision-makers  but  were  then  later  applied  to  the  normalized  values  of  the  characteristics  to  obtain 
MOEs  (personal  correspondence  with  DoD  officials). 
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budgets,  affordability,  or  funding  (resource  constraints) .  As  a  consequence,  the 
problem  is  typically  not  structured  as  described  above. 

Instead  of  structuring  an  AoA  as  a  constrained  optimization,  a  popular 
decision-analysis  approach  is  (towards  the  end  of  an  AoA)  to  simply  attach  a  weight 
to  cost  and  introduce  it  directly  into  the  utility  function. “  As  opposed  to  a  benefit/cost 
(or  effectiveness/cost)  ratio,  this  popular  MCDM  approach  generates  an  overall 
“value”  function  that  is  essentially  a  weighted  average  of  cost  and  effectiveness. 

In  effect,  alternatives  are  ranked  and  selections  are  made  by  Decision- 
Makers  through  an  unconstrained  optimization  shaped  by  analysts,  where  the  best 
alternative  is  the  one  that  maximizes  the  overall  effectiveness  or  value  function  V  = 
V(MOE;  COST).  “Deterministic  decision  analysis  is  concerned  with  finding  the  most 
preferred  alternative  in  decision  space  by  constructing  a  value  function  representing 
a  decision  maker’s  preference  structure,  and  then  using  the  value  function  to  identify 
the  most  preferred  solution”  (Ramesh  &  Zionts,  1997,  p.  421).  The  linear,  additive, 
separable  version  of  this  value  function  is  frequently  used  to  calculate  a  positively 
weighted  MOE  and  negatively  weighted  cost  for  each  alternative.  For  example,  see 
Beil  &  Wein  (2003),  Che  (1993),  Clemen  (1996),  Kirkwood  (1997),  French  (1986), 
Keeney  &  Raiffa  (1976),  Keeney  (1994),  Flwang  and  Yoon  (1981),  Liberatore  (1987), 
Pinker,  Samuel,  and  Batcher  (1995),  Varzsonyi  (1995),  etc. 

The  typical  decision  sciences’  (MCDM)  approach  to  an  AoA^^  involves: 

Given  several  alternatives,  select  the  preferred  alternative  that  provides  the 

best  value,  or  maximizes:  V(MOE,COST)  =  w1*MOE  -  w2*COST 


“In  the  European  Union,  a  legislative  package  intended  to  simplify  and  modernize  existing  public 
procurement  laws  was  recently  adopted.  As  before,  the  new  law  allows  for  two  different  award 
criteria:  lowest  cost  and  best  economic  value.  The  new  provisions  require  that  the  procurement 
authority  publishes  ex-ante  the  relative  weighting  of  each  criteria  used  when  best  economic  value  is 
the  basis  for  the  award”  (see  EC,  2004a;  2004b). 

“An  AoA  is  an  analytical  Comparison  of  the  operational  Effectiveness,  suitability,  and  Life-Cycle 
Cost  of  Alternatives  that  satisfy  established  Capability  needs”  (DoD,  2006,  July  7). 
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This  requires  two  important  modeling  efforts:  1)  building  an  effectiveness  (MOE) 
model  (non-cost  factors;  performance  =  quality,  schedule,  etc.);  and  2)  building  a 
cost  model  (costs/prices;  estimate  total  system  lifecycle  costs,  total  ownership 
costs).  Once  these  independent  modeling  efforts  are  completed,  the  overwhelming 
challenge  is  to  integrate  the  two  by  using  either  benefit/cost  ratios  (discussed  earlier) 
or  by  assigning  a  relative  weight  to  cost  (a  value  for  w2  in  the  example  above). 

The  typical  recommendation  in  the  applied  literature  to  integrate  cost  and 
effectiveness  is  for  analysts  to  ask  the  DM:  “How  important  is  cost  relative  to 
effectiveness?”  The  Federal  Acquisition  Regulation  (FAR)  supports  this  approach: 
“The  solicitation  shall  state  whether  all  evaluation  factors  other  than  cost/price,  when 
combined  [i.e.,  MOE],  are  significantly  more  important  than,  approximately  equal  to, 
or  significantly  less  important  than  cost/price”  (General  Services  Administration, 
2005,  March,  Section  15.101-1(2)).  The  Office  of  Management  and  Budget  (OMB)^^ 
also  promotes  this  approach:  “The  specific  weight  given  to  cost  or  price  shall  be  at 
least  equal  to  all  other  evaluation  factors  combined  unless  quantifiable  performance 
measures  can  be  used  to  assess  value  and  can  be  independently  evaluated”  (0MB, 
2003,  p.  B-8).  The  decision  regarding  the  appropriate  relative  weight  to  assign  to 
costs  is  a  hotly  debated  and  contentious  issue  that  has  resulted  in  bid  protests  being 
sustained  by  the  GAO.  (See  Melese  et.  al.  2010) 

A  key  proponent  of  this  (MCDM)  decision  methodology  offers  an  example  of 
administrators  who  are  evaluating  alternative  pollution  control  devices  being  asked 
to  answer  questions  such  as  “Which  is  more  important,  costs  or  pollutant 
concentrations?”  (Keeney,  1994,  p.  797).  As  the  author  is  quick  to  point  out,  the 


According  to  the  FAR,  “source  selection”  is  the  decision  process  used  in  competitive,  negotiated 
contracting  to  select  the  proposal  that  offers  the  “Best  Value”  to  the  government.  “In  different  types  of 
acquisition,  the  relative  importance  of  cost  or  price  may  vary”  (General  Services  Administration,  2005, 
Section  15.101).  In  describing  some  lessons  learned,  Gansler  (2003)  recommended:  Use 
performance-based  contracting;  Do  not  list  tasks  [mix  of  inputs],  instead  state  results  sought  or 
problems  to  be  solved  [desired  attributes/characteristics  of  outputs/outcomes];  Choose  contractors 
according  to  “Best  Value”;  In  the  source  selection,  trade  off  performance  and  price  instead  of  simply 
awarding  to  the  lowest  bidder  (p.  15). 
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problem  with  this  approach  is  that  without  some  estimate  of  the  total  budget 
available  or  any  knowledge  of  opportunity  costs,  one  cannot  expect  the  DM  to 
provide  a  sensible  answer.  Ironically,  that  same  author-one  of  the  strongest 
proponents  of  this  methodology-warns:  “I  personally  do  not  want  some 
administrator  to  give  two  minutes  of  thought  to  the  matter  and  state  that  pollutant 
concentrations  are  three  times  as  important  as  cosP^  (Keeney,  1994,  p.  797). 

Figures  2  and  3  offer  an  illustration.  Figure  2  reflects  a  situation  in  which  the 
decision-maker  believes  costs  to  be  important  enough  (and  thus  assigns  a 
sufficiently  large  relative  weight,  w2/w1,  to  cost)  that  the  preferred  alternative  is  A1 
(the  low-cost  option).  The  opposite  case  is  illustrated  in  Figure  3.^* 

How  does  a  decision-maker  (DM)  decide  on  an  appropriate  weight  to  assign 
to  costs?  Consider  an  extreme  case.  Suppose  affordability  is  not  an  issue,  so 
funding  is  not  an  issue.  In  that  case,  the  budget  is  not  binding,  making  costs 
irrelevant.  Clearly,  this  means  a  zero  weight  should  be  assigned  to  costs,  and  the 
alternatives  can  be  ranked  exclusively  on  the  basis  of  their  MOEs  (e.g.,  A2  wins).  As 
a  consequence,  any  weight  (w2)  applied  to  costs  must  reflect  an  implicit  concern 
about  affordability  (budgets/funding  levels). 

A  key  hypothesis  in  the  Economic  Evaluation  of  Alternatives  (EEoA)  is  that  if 
a  DM  pays  any  attention  to  costs  (i.e.,  places  a  weight  on  cost),  it  is  because  he/she 
acknowledges  an  implicit  affordability  or  budget  constraint  or  recognizes  there  may 
be  some  opportunity  cost  for  funds  committed  to  the  program.  This  is  directly  related 


“  Surprisingly,  the  author  has  continued  to  write  prolifically  in  this  field  and  continued  to  promote  this 
decision  criterion,  apparently  never  taking  the  time  to  reflect  back  on  these  key  observations. 

Note  that  the  slope  of  the  straight-line  indifference  curves  reflect  the  DM’s  relative  preference  (or 
trade-offs).  These  trade-offs  between  MOE  and  Cost  are  given  by  -w2/w1 . 
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to  the  higher-level  affordability  discussions  in  Section  1  that  included  Requirements 
Generation,  Defense  Acquisition,  and  PPBE.“ 


So  Which  Alternative  is  “Best”? 
Decision  Sciences  Approach 


Max  V  =  V(MOE,Cost)  =  w1*MOE  -  w2*Cost 

Ask  Decision  Maker  What  is  More  Important:  MOE  or  Cost? 

(dMOE/dCost  =  w2/w1) 

MOE(Utils) 

c/3 
C/3 

(D 

<D 

> 


O 


If  Cost  has  a  sufficiently 
greater  weight  (w2»w1 ), 
then  low  cost,  low 
effectiveness  alternative 
Ai  wins 


Cost 


Figure  2.  When  Cost  Is  Relatively  More  Important  than  Effectiveness 


The  Army’s  Economic  Analysis  (EA)  Manual  states  that  “[a]  good  EA  should  go  beyond  the 
decision-making  process  and  become  an  integral  part  of  developing  requirements  in  the  PPBE 
process”  (DoA,  2001,  February,  p.  12). 
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Which  Alternative  is  “Best”? 
Decision  Sciences  Approach 

Max  V  =  V(MOE,Cost)  =  w1*MOE  -  w2*Cost 


MOE(Utils) 


c/5 

C/5 

(D 

a 

<D 

> 
•  ^ 

-4— > 

o 

W 


(dMOE/dCost  =  w2/w1) 


0  A2 


Q  A1 


If  Performance  has  a 
sufficiently  greater  weight 

(w1»w2),  then  high  cost, 
high  effectiveness 
alternative  A2  wins 


Cost 


Figure  3.  When  Effectiveness  Is  Relatively  More  Important  than  Cost 

The  irony,  as  Keeney  (1994)  rightly  observed,  is  that  to  assign  any  weight  to 
costs  requires  the  DM  to  have  some  understanding  of  affordability  (why  would  you 
care  about  costs  unless  there  are  constraints  on  the  budget/funding/resources 
available?)  and  an  appreciation  of  relevant  opportunity  costs.  But  if  this  information 
is  known,  then  analysts  and  decision-makers  have  no  reason  to  take  the  MCDM 
approach  and  assign  a  weight  to  costs  since  the  more  robust,  constrained- 
optimization  (mathematical  programming)  EEoA  approach  is  available. 

In  fact,  it  is  relatively  straightforward  to  demonstrate  that  even  if  the  DM  had 
perfect  information  about  the  budget  (funding/affordability)  and  attempted  to  interpret 
that  information  through  a  weight  assigned  to  the  relative  cost  (price)  of  alternatives 
(as  illustrated  in  Figures  2  and  3),  the  rankings  that  resulted  would  only 
coincidentally  correspond  to  rankings  obtained  under  the  full-information- 
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constrained-optimization  EEoA  (in  which  MOE  is  maximized  subject  to  the  budget 
constraint).^® 

This  is  a  damning  result  that  clearly  undermines  the  way  MCDM  is  typically 
applied  by  analysts  to  support  AoAs.  If  there  is  no  guarantee  that  the  MCDM 
approach  will  yield  consistent  results  under  full  information  (including  affordability), 
then  using  this  criterion  with  less-than-perfect  information  (i.e.,  in  the  absence  of 
explicit  assumptions  about  affordability/budgets/funding)  is  clearly  problematic.  In 
fact,  the  GAO  emphasizes  that  “[wjith  high  levels  of  uncertainty  [...]  funding  needs 
are  often  understated”  (GAO,  2009,  p.  9).  A  very  real  risk  in  applying  the  MCDM 
approach  is  that  if  AoAs  “fail  to  balance  needs  with  resources  [funding/budgets]  [...] 
un-executable  programs  [are  allowed]  to  move  forward,  [and]  program  managers 
[...]  are  handed  [...]  a  low  probability  of  success”  (GAO,  2009,  p.  10). 

In  conclusion,  the  popular  MCDM  decision  sciences  approaches  that 
underpin  most  AoAs  either:  i)  explicitly  ignore  affordability  in  applying  the 
benefit/cost  (effectiveness/cost)  ratio  criterion,®’’  or  ii)  implicitly  attempt  to  capture 
affordability  by  choosing  a  relative  weight  to  assign  to  cost,  maximizing  gains  and 
minimizing  costs  in  a  value  function  such  as:  Maximize  V  =  V(MOE,Cost)  =  w1*MOE 
-  w2*Cost.®®  Again,  to  quote  Hitch  and  McKean  (1967): 


®®  The  weight  on  cost  in  the  unconstrained  optimization  (MCDM  approach)  roughly  corresponds  to  the 
Lagrangian  multiplier  (shadow  price)  of  the  budget  constraint  in  the  constrained  optimization  (the 
EEoA  approach). 

“One  common  ‘compromise  criteria’  is  to  pick  that  [alternative]  which  has  the  highest  ratio  of 
effectiveness  to  cost.  [Mjaximizing  this  ratio  is  the  [decision]  criterion.  [While]  it  may  be  a  plausible 
criterion  at  first  glance  [...]  it  allows  the  absolute  magnitude  of  [effectiveness]  or  cost  to  roam  at  will. 

In  fact,  the  only  way  to  know  what  such  a  ratio  really  means  is  to  tighten  the  constraint  until  either  a 
single  budget  (or  particular  degree  of  effectiveness)  is  specified”  (Hitch  &  McKean  pp.  1 65-167). 

®®  In  a  section  describing  Building  a  Model,  Fisher  (1965)  comments:  “Since  by  definition  a  model  is 
an  abstraction  from  reality,  the  model  must  be  built  on  a  set  of  assumptions.  These  assumptions  must 
be  made  explicit.  If  they  are  not,  this  is  to  be  regarded  as  a  defect  of  the  model  design”  (p.  190).  It  is 
easy  to  conceal  the  importance  of  affordability  (budget/funding)  issues  in  the  MCDM,  decision 
sciences  approach  that  underpins  many  AoAs.  In  sharp  contrast,  the  Economic  Evaluation  of 
Alternatives  approach  encourages  explicit  affordability  (budget/funding)  assumptions. 
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One  ubiquitous  source  of  confusion  is  the  attempt  to  maximize  gain 
[w1*MOE]  while  minimizing  cost  [w2*Cost]  [...].  If  a  person  approaches  a 
problem  with  the  intention  of  using  such  a  [decision]  criterion,  he  is  confused 
to  begin  with  [...].  [A]  criterion  in  which  the  budget  [affordability]  [...]  is 
specified  has  the  virtue  of  being  aboveboard,  (pp.  165-167) 

Rather  than  attempt  to  get  a  DM  to  implicitly  reveal  his  or  her  affordability 
concerns  through  a  weight  assigned  to  costs  (prices)  of  alternatives,  the  Economic 
Evaluation  of  Alternatives  (EEoA)  recommends  an  explicit,  transparent  and 
accountable  approach:  treat  cost  as  an  independent  variable  (CAIV).  The  relevant 
CAIV  concept  applied  here  follows  a  definition  posted  on  the  Office  of  the  Under 
Secretary  of  Defense  (Acquisition  &  Technology)  website  in  early  1999.  It  states  that 
CAIV  is:  “DoD’s  acquisition  methodology  [...]  making  [...]  performance  a  function  of 
available  budgeted  resources  ’  (see  Lorell  &  Graser,  2001,  p.  33).  The  Office  of 
Management  and  Budget  (0MB)  Circular  A-1 09  for  Major  Systems  Acquisition 
mentions  the  goal  of  “design-to-cost.”  According  to  the  Defense  Acquisition 
Guidebook,  “all  participants  [...]  are  expected  to  recognize  the  reality  of  fiscal 
constraints”  (DoD,  2006,  July  7,  Section  3.2.4).  It  is  clear  from  these  examples  that 
CAIV  is  directly  concerned  with  affordability.^®  The  next  section  illustrates  how 
affordability  is  explicitly  incorporated  in  the  Economic  Evaluation  of  Alternatives 
(EEoA). 


Under  CAIV  performance  and  schedule  are  considered  dependent  on  the  funds  available  for  a 
specific  program.  “Using  affordability  as  the  key  criterion,  the  service  headquarters  divides  a  fixed 
budget  among  competing  programs.  Here  the  cost  goals  are  used  in  developing  budget  required  for 
that  program  and  compared  with  the  available  dollars  in  the  Program  Objective  Memorandum  (POM) 
years  based  on  the  priority  level  established  by  the  service,  Joint  Requirements  Oversight  Council 
(JROC),  and  others.  This  fixed  budget  based  on  the  priority  of  the  program  is  the  reality  of  what  is 
available  for  structuring  the  program.”  (Rush  1997  p.167) 
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V.  Six  Ways  to  Structure  an  Economic  Evaiuation 
of  Aiternatives  (EEoA) 


There  are  six  ways  that  analysts  and  decision-makers  can  structure  a 
deterministic  Economic  Evaluation  of  Alternatives  (EEoA).  The  first,  third,  and  fourth 
approaches  are  in  the  spirit  of  cost  as  an  independent  variable  (CAIV). 

It  is  also  useful  to  distinguish  between:  i)  intra-program  analysis  approaches 
(#1-#5),  and  ii)  the  inter-program  analysis  approach  (#6).  In  the  case  of  intra¬ 
program  analysis,  the  decision-maker  (DM)  associated  with  the  program  is  assumed 
to  have  sufficient  information  to  select  an  alternative  without  reference  to  competing 
programs.  That  is  not  the  case  in  inter-program  analysis,  which  requires  an  explicit 
opportunity-cost  approach.  The  six  EEoA  approaches  appear  in  Table  1 . 

Table  1.  Six  Approaches  to  Structure  an  EEoA 

I.  INTRA-PROGRAM  ANALYSIS 

A)  Build  Alternatives 

1.  Fixed-budget  Approach 

2.  Fixed-effectiveness  Approach 

3.  Expansion-path  Approach  (Construct  alternatives  as  Cost- 
output/Effectiveness  Relations  or  Response  Functions:  Multi¬ 
stage  Micro-economic  Production  Model) 

B)  Modify  Existing  Alternatives:  Level  the  Playing  Field 

4.  Modified-budget  Approach:  GOTO  1 

5.  Modified-effectiveness  Approach:  GOTO  2 

II.  INTER-PROGRAM  ANALYSIS 

6.  Opportunity  Cost/Benefit  Approach 

There  are  two  possibilities  highlighted  within  the  Intra-program  analysis 
approach  outlined  in  Table  1.  The  first  possibility  is  that  DMs  (and/or  analysts)  are 
able  to  construct/define/build  alternatives  (endogenous  alternatives).  The  second 
possibility  is  that  alternatives  are  already  constructed/defined/built  (pre-specified) 
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and  must  simply  be  evaluated  (exogenous  alternatives).  This  section  describes  each 
of  the  six  EEoA  approaches  in  some  detail.'"’ 

An  earlier  quote  from  Hitch  and  McKean  (1967)  highlights  the  first  two  EEoA 
approaches:  “[A]  criterion  in  which  the  budget  or  level  of  effectiveness  is  specified 
has  the  virtue  of  being  aboveboard”  (p.  167).  Beginning  with  the  Fixed-budget 
Approach  (#1),  it  is  useful  to  recall  another  quote  from  Hitch  and  McKean  (1967); 
“The  test  of  maximum  effectiveness  fora  given  budget  seems  much  less  likely  to 
mislead  the  unwary”  (p.  167). 

1.  Fixed-budget  Approach. 

In  his  groundbreaking  book  Cost  Considerations  in  Systems  Analysis,  Fisher 
(1971)  describes  the  first  approach  to  EEoA: 

In  the  fixed  budget  case,  the  alternatives  being  considered  are  compared  on 
the  basis  of  effectiveness  likely  to  be  attainable  for  the  specified  budget  level” 
(p.  12).  “The  analysis  attempts  to  determine  that  alternative  (or  feasible 
combination...)  which  is  likely  to  produce  the  highest  effectiveness”  (p.  10). 

In  a  footnote,  Fisher  (1971)  adds:  “the  fixed  budget  situation  is  somewhat  analogous 
to  the  economic  theory  of  consumer  [optimization]  [...].  For  a  given  level  of  income 
[budget/funding]  the  consumer  is  assumed  to  behave  in  such  a  way  that  he 
maximizes  his  utility”  (p.  10).  Drawing  on  these  observations,  the  Fixed-budget 
Approach  to  the  EEoA  leverages  Lancaster’s  “characteristics  approach  to  demand 
theory”  (1966a;  1966b;  1971;  1979).  Synthesizing  the  work  of  Gorman  (1980), 

Stigler  (1945),  Theil  (1952),  and  others  (that  also  provided  the  early  foundations  of 
the  MCDM  literature),  Lancaster  offers  economists  (and  defense  analysts)  a  familiar 
way  to  analyze  the  consumer’s  (defense  decision-maker’s)  choice  problem  (i.e.,  to 
choose  among  alternative  defense  investments). 


A  separate  paper  available  upon  request  describes  the  static,  deterministic,  multi-stage, 
constrained  optimization,  micro-economic  production  (procurement  auction)  model  that  underpins  the 
third,  and  most  general,  approach  to  the  EEoA:  the  Expansion-path  Approach. 
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In  Lancaster’s  model,  different  vendors  generate  different  bundles  of 
characteristics  evaluated  by  decision-makers  (consumers).  Lancaster’s  model 
proposes  that  to  choose  among  alternative  bundles  of  commodities  (say  computers), 
defense  decision-makers  maximize  their  utility  function,  which  is  defined  over  a 
desired  set  of  multiple  criteria,  attributes,  or  characteristics  and  is  subject  to  a  budget 
[funding/affordability]  constraint.'^^  In  this  approach,  the  cost-effective  alternative  is 
the  one  that,  for  a  given  budget  (funding/expenditure/affordability  level),  generates 
the  best  mix  of  characteristics  evaluated  using  the  decision-maker’s  utility  function. 


Cost-Effectiveness  EEoA 

Build  Alternatives 


1.  Fixed  Budget  Approach 

Maxinnize  Effectiveness  subject  to  Budget  Constraint 
(construct  alternatives  for  given  budget) 


Outsourcing  Opportunity: 

Can  we  get  more  bang  for  the 
same  bucks? 


Benefit/Cost 


Figure  4.  Fixed-budget  Approach 


Note  that  we  refer  to  the  usual  deterministic  “utility  function”  that  is  conventional  in  the  economics 
literature.  This  is  in  contrast  to  the  way  a  utility  function  is  typically  defined  in  the  decision  sciences 
and  operations  management  literature — that  is,  as  a  stochastic  function.  The  value  function  described 
in  the  latter  literature  is  similar  to  our  utility  function,  except  that  costs  can  enter  into  a  value  function 
and  are  excluded  from  our  utility  function  since  they  appear  as  part  of  the  budget  constraint. 
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The  Fixed-budget  Approach  illustrated  in  Figure  4  is  the  first  of  six  ways 
proposed  to  structure  an  EEoA.  In  Figure  4,  the  budget  (funding/affordability  level) 
estimate  for  the  program  is  set  at  B*.  The  three  vendor  alternatives  constructed 
given  this  budget  are  A1,  A2,  and  A3.  Given  its  superior  performance  in  terms  of  its 
MOE,  vendor  A3  wins  the  competition,  which,  in  this  case,  is  also  determined  by  the 
highest  benefit/cost  ratio.'^^ 

2.  Fixed-effectiveness  Approach 

The  second  way  to  structure  an  EEoA  is  (mathematically)  the  dual  of  the  first: 
minimize  the  cost  of  achieving  a  given  MOE.  RAND  Corporation’s  AoA  for  the  KC- 
135  Recapitalization  adopts  this  approach:  “[l]n  this  AoA,  the  most  ‘cost-effective’ 
alternative  [fleet]  means  precisely  the  alternative  whose  effectiveness  meets  the 
aerial  refueling  requirement  at  the  lowest  cosf  (Kennedy  et  al.,  2006,  p.  7,  emphasis 
added).  Figure  5  offers  an  illustration  in  which  the  lowest  cost  option  for  a  fixed  MOE 
is  Al ,  and  it  also  offers  the  highest  benefit/cost  ratio. 

Another  example  is  provided  by  a  section  on  cost-effectiveness  analysis  in 
0MB  Circular  A-94  that  states:  “A  program  is  cost-effective  if,  on  the  basis  of  life 
cycle  cost  analysis  of  competing  alternatives,  it  is  determined  to  have  the  lowest 
costs  [...]  for  a  given  amount  of  benefits  [...].  Cost-effectiveness  analysis  can  also  be 
used  to  compare  programs  with  identical  costs  [budgets/funding]  but  differing 
benefits”  (0MB,  1992,  October  29,  p.  4).  The  latter  part  of  the  quote  refers  to  the  first 
approach  to  structuring  an  EEoA,  and  the  former  to  the  second  approach. 


Note  that  in  the  first  and  second  EEoA  approaches,  since  either  the  budget  (funding  level)  or  MOE 
(level  of  effectiveness)  is  anchored  in  the  constrained  optimization,  the  benefit/cost  ratio  decision 
criterion  can  be  used  as  a  decision  rule  in  the  selection  process.  The  steeper  the  slope  from  the  origin 
through  an  alternative  (A1,  A2,  A3),  the  bigger  the  bang-for-the-buck. 
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Cost-Effectiveness  EEoA 

Build  Alternatives 


2.  Fixed  Effectiveness  Approach 

Dual:  Minimize  Costs  subject  to  Effectiveness  Constraint 
(construct  alternatives  for  given  MOE) 


MOE;(Utils) 


MOE^ 


w 

w 

LU 
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Benefit/Cost 
=  MOE*/$Bi 
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Outsourcing  Opportunity: 

Can  we  spend  less  bucks  for 

the  same  bang?  omb 
Circular  A-76:  Statement  of 
Work(SOW);  Invitation  for  Bid(IFB) 


If  A1  is  the  In-house  MEO,  then 
“Outsourcing’s  Out  &  Insourcing’s  In” 

(DoD  FY  2005  :  71%  A-76  won  by  in-house  MEO) 


Cost($) 


Figure  5.  Fixed -effectiveness  Approach 

Another  example  of  the  Fixed-effectiveness  Approach  to  structuring  an  EEoA 
is  given  by  public-private  (competitive  sourcing)  competitions  conducted  under  OMB 
Circular  A-76,  which  “requires  [...]  a  structured  process  for  [evaluating]  the  most 
efficient  and  cost-effective  method  of  performance  for  commercial  activities”  (2003, 
May  29,  Attachment  B).  This  involves  four  steps:  1 )  develop  a  Statement  of  Work 
(SOW)  or  Performance  Work  Statement  (PWS)  to  define  desired 
performance/effectiveness,  2)  construct  the  Most  Efficient  Organization  (MEO)  for 
the  in-house  competitor,  3)  issue  an  Invitation  for  Bid  (IFB)  for  well-defined,  routine 
commercial  activities  (SOW  or  PWS),  and  4)  compare  bids  or  proposals  (source 
selection)  and  select  the  “least  cost”  for  IFB. 
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Finally,  Title  10,  Subtitle  A,  Part  IV,  Chaper  146,  Section  2462  of  the  US  Code 


reads: 


A  function  of  the  Department  of  Defense  [...]  may  not  be  converted  [...]  to 
performance  by  a  contractor  unless  the  conversion  is  based  on  the  results  of 
a  public-private  competition  that  [...]  examines  the  cost  of  performance  of 
the  function  by  Department  of  Defense  civiiian  empioyees  and  the  cost 
of  performance  of  the  function  by  one  or  more  contractors  to 
demonstrate  whether  converting  to  performance  by  a  contractor  will  result  in 
savings  to  the  Government  over  the  life  of  the  contract.  (2007,  January  3, 
emphasis  added) 

This  law  concerning  public-private  competitions  offers  another  illustration  of 
the  Fixed-effectiveness  Approach,  minimizing  the  cost  of  achieving  a  given  MOE. 

3.  Expansion-path  (Response-function)  Approach 

In  a  quote  cited  above.  Hitch  and  McKean  (1967)  strongly  hint  at  the  third 
approach  to  structuring  an  EEoA: 

The  test  of  maximum  effectiveness  for  a  given  budget  seems  much  less 
likely  to  mislead  the  unwary”  [. . .].  “As  a  starter,  [. . .]  severai  budget  sizes 
can  be  assumed.  If  the  same  [alternative]  is  preferred  for  all  [...]  budgets, 
that  system  is  dominant.  If  the  same  [alternative]  is  not  dominant,  the  use  of 
several  [...]  budgets  is  nevertheless  an  essential  step,  because  it  provides 
vital  information  to  the  decision  maker,  (pp.  167,  176,  emphasis  added) 

This  third  way  to  structure  an  EEoA  provides  a  foundation  for  all  the  others.  It 
is  described  here,  and  is  modeled  mathematically  in  a  companion  paper  (available 
upon  request).  The  model  involves  a  three-step  process  that  includes  multiple 
players. 

For  ease  of  exposition,  we  assume  three  players:  the  military  buyer  and  two 
competing,  private  vendors.  The  first  step  is  for  the  military  buyer  to  issue  a 
solicitation  in  the  form  of  an  Economic  Request  for  Proposals  (ERFP). 

An  ERFP  (solicitation)  states  all  significant  non-price  factors 
(criteria/attributes/characteristics)  that  the  agency  expects  to  consider  in  evaluating 
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proposals,  but  also  includes  an  affordability  assessment — e.g.,  optimistic  (B1), 
pessimistic  (B2)  and  most-likely  budget  (funding)  estimates  for  the  overall  program. 

Once  a  solicitation  is  issued  in  the  form  of  an  ERFP,  interested  vendors 
submit  their  offers  based  on  the  different  budget  scenarios  (see  Figure  6)  and  the 
selection  process  begins.'*^  Each  vendor  responds  by  offering  the  best  combination 
(bundle)  of  attributes  (non-price  factors)  they  can  produce  at  each  funding  level, 
given  their  individual  production  and  cost  functions.  Assuming  the  award  is  made 
without  discussions  (pursuant  to  FAR  52.212-1  and  52.215-1),  the  military  buyer 
then  employs  a  “secret”  scoring  rule  to  rank  vendors.'*'* 

ECONOMIC  APPROACH:  Endogenous  Alternatives  (“Engel  Curves”) 

3.  Expansion  Path  (Response  Function)  Approach 
(Alternatives  are  Cost-Effectiveness  Relations,  not  Points) 

Explore  impact  of  budget  cuts  (Identify  vendor  responses) 


Source  Selection  Decision:  A2  for  pessimistic  budget;  A1  for  optimistic  budget 


Figure  6.  Expansion  path  Approach 


'*^  The  budget  announcements  are  analogous  to  an  agency  exploring  in  order  to  uncover  its  true 
“reservation  price”  for  the  acquisition  (given  the  competing  demands  for  scarce  budgets).  The 
adoption  of  this  approach  of  evaluating  vendor  proposals  under  different  reservation  prices  could 
eventually  lead  to  greater  use  of  fixed-price  contracts. 

'*'*  The  buyer  reveals  the  desired  attributes/characteristics  of  the  investment  to  the  sellers — but  not  the 
weights — and  requests  a  single  offer  from  each  seller  for  a  pre-specified  budget  (affordability) 
constraint,  and  then  the  buyer  chooses  the  one  he  prefers  among  the  submitted  offers.  “We  call  this 
procedure  a  ‘single-bid  auction  with  secret  scoring  rule’”  (Asker  &  Cantillon,  2004,  p.  1). 
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Since  each  vendor  is  assumed  to  have  different  production  and  cost  functions 
to  generate  the  desired  attributes,  the  vendors’  constrained  optimizations  define 
distinct  response  functions  (expansion  paths),  one  for  each  vendor/®  The  interesting 
case  is  where  there  is  a  crossover.  For  example,  at  the  lower  budget  B2,  Alternative 
2  would  be  selected  as  offering  the  highest  MOE,  whereas  with  a  higher  budget  B1 
we  would  switch  to  Alternative  1 .  So  the  highest  MOE  attainable  at  each  budget 
level  is  represented  by  the  two  light  (yellow)  points.  (See  Figure  6)  The  model  is 
briefly  described  in  Table  2  below. 

Table  2.  Economic  Evaluation  of  Alternatives  (Approach  #3): 


Military  Buyer  Goal:  Select  an  alternative  that  maximizes 
MOE  =  utility  function  =  U(non-cost  factors/attributes). 

Subject  to  BUDGET  constraint 

Vendor  Goal:  Select  a  mix  of  non-cost  factors  that 

maximizes  Q  =  Production  Function  =  Q(non-cost  factors/attributes) 

Subject  to  Sum  of  Costs  of  Attributes  =  c1  x  a1  +  c2  x  a2  +...  <  =  Budget 

Military  Buyer  Requirements 

MOE:  build-effectiveness  model  (non-cost  factors:  Performance  =  quality, 
schedule,  etc.) 

COST:  build-cost  model  (costs/prices:  Estimate  total  system  lifecycle  costs, 
total  ownership  costs) 

AFFORDABILITY:  Estimate  budget  (forecast  funding  available  for  the 
program:  e.g.  optimistic,  pessimistic  and  most  likely) 

Private  Vendor  Requirements 

Production  Function:  Possible  attribute  mixes  given  vendor-specific 
technology 

Total  Costs:  Vendor-specific  costs  of  producing  each  attribute 
Vendor  Proposal  constructed  as  a  function  of  Buyer’s  Budget  constraint 


From  the  Envelope  Theorem,  the  Lagrangian  multiplier  in  each  vendor’s  constrained  optimization 
problem  reveals  the  marginal  product  (the  extra  output  or  attribute  mix  they  are  capable  of  producing) 
as  the  military  buyer  (the  DoD)  relaxes  or  tightens  its  funding  constraint  for  the  overall  program. 
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Expansion  paths  exist  for  each  vendor  that  reveal  the  combination  of 
attributes  each  vendor  can  offer  at  different  budget  levels  (e.g.,  pessimistic, 
optimistic,  and  most-likely). 

This  third  fundamental  EEoA  approach  follows  our  earlier  quote  from  Hitch 
and  McKean  (1967)  that  recommends: 

As  a  starter  [...]  several  budget  sizes  can  be  assumed.  If  the  same 
[alternative]  is  preferred  for  all  [...]  budgets,  that  system  is  dominant  [...].  If 
the  same  [alternative]  is  not  dominant  the  use  of  several  [...]  budgets  is 
nevertheless  an  essential  step,  because  it  provides  vital  information  to  the 
decision  maker,  (p.  176) 

This  is  illustrated  in  Figure  6  with  two  notional  budget  levels:  B1  (pessimistic) 
and  B2  (optimistic).  This  three-stage  procurement  process  is  summarized  in  Table  3. 

Table  3.  Three-stage  Multiattribute  Procurement  Auction  (Expansion- 

path  Approach) 


1)  First  Stage:  (CAIV) 

-  The  DoD  provides  notionai  budget  guidance  (B)  to  alternative 
vendors  for  the  program.  The  DoD  searches  for  the  optimum 
product  (Procurement)  and/or  service  (R&D;  O&M)  package  that  it 
can  obtain  at  that  price,  B.  The  DoD  also  reveals  optimistic  and 
pessimistic  budget  guidance. 

-  The  DoD  defines  the  set  of  characteristics/attributes  that  it 
vaiues,  and  this  is  known  to  vendors.  However,  the  DoD’s  precise 
utility  function  over  those  characteristics  is  unknown  to  vendors 
(“secret”  scoring  rule). 

2)  Second  Stage:  (Target  Costing) 

-  Vendors  have  different  costs  and  production  functions  for 
generating  products  or  services  (defined  as  bundles  of 
characteristics). 

-  Each  vendor  maximizes  its  output  offer  (an  optimal  mix  of  the 
desired  characteristics)  subject  to  their  particular  budget 
constraint  (which  includes  the  DoD’s  budget  guidance  and  the 
vendor’s  individual  costs  to  produce  a  unit  of  each  characteristic). 

-  This  is  the  product  and/or  service  package  (output)  a 
particuiar  vendor  is  abie  to  propose  for  each  possibie  budget 
(B)  given  their  production  function  (technical  production 
possibilities)  and  their  vendor-specific  costs  of  generating  those 
characteristics. 
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3)  Third  Stage:  (Selection) 

-  With  the  latest  budget  forecast,  the  DoD  selects  from  among  the 
optimized  characteristic  bundles  proposed  by  each  vendor  the 
bundle/alternative  (total  product/service  package)  that  maximizes 
the  DoD’s  utiiity  function. 

In  Figure  6,  each  vendor’s  expansion  paths  are  transformed  (through  the 
government’s  utility  function)  into  cost-utility  or  cost-effectiveness  response  functions 
(A1  and  A2).  These  response  functions  reveal  each  vendor’s  proposals  under 
different  budget  scenarios  and  represent  the  most  general  definition  of  “alternatives” 
in  the  Economic  Evaluation  of  Alternatives  (EEoA).  Given  a  range  of  likely  budgets 
for  the  program,  the  most  effective  vendor  over  that  range  of  budgets  can  be 
selected  by  the  buyer.  This  approach  explicitly  addresses  a  key  concern  voiced  by 
the  GAO: 

A  cost  estimate  is  [...]  usually  presented  to  decision  makers  as  a  [...]  point 
estimate  that  is  expected  to  represent  the  most  likely  cost  of  the  program  but 
provides  no  information  about  the  range  of  risk  and  uncertainty  or  level  of 
confidence  associated  with  the  estimate.  (GAO,  2009,  p.  9) 

Suppose  the  planning  process  allows  the  optimistic  budget  assumption,  B1, 
to  be  used  by  vendors  in  preparing  proposals.  If  this  is  what  the  actual  budget  turns 
out  to  be,  then  vendor  A1  would  be  selected.  If  the  PPBE  process  ultimately  narrows 
the  budget/expenditure  constraint  for  the  program  to  B2,  then  Alternative  2  is 
superior.  However,  if  vendor  proposals  were  only  solicited  for  the  optimistic  budget 
level,  B1,  then  when  reality  strikes  and  budget  B2  is  all  that  is  available  for  the 
program,  choosing  vendor  proposal  A1  could  turn  out  to  be  a  significant  error  (see 
Figure  6).  This  discussion  reveals  how  the  EEoA  approach  relies  upon  and 
reinforces  the  importance  of  iterative  interactions  between  the  requirements 
generation  system,  defense  acquisition  system  and  PPBE  process. 
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The  first  three  ways  to  structure  an  EEoA  are  for  cases  when  alternatives  can 
be  constructed/built/designed  (i.e.  endogenous  alternatives).  The  last  three  are 
appropriate  when  alternatives  are  exogenously  determined  (or  pre-specified).  The 
most  interesting  analysis  in  the  latter  case  occurs  when  one  alternative  costs  more 
but  offers  greater  utility  (MOE)  and  when  others  cost  less  and  offer  less  utility 
(MOE). 

4.  Modified  Budget  Approach 

Consider  the  case  where  alternatives  are  provided  that  have  been  developed 
exogenously — for  example,  on  the  basis  of  a  manpower  or  squadron  constraint  (e.g., 
one  computer  per  person,  or  a  certain  number  of  aircraft  per  squadron).  If  the  overall 
budget  or  desired  level  of  effectiveness  (MOE)  for  a  program  is  not  available  (and 
analysts  and  DMs  have  not  structured  the  problem  in  terms  of  affordability),  then  it  is 
likely  these  pre-specified  alternatives  solicited  from  different  vendors  have  different 
costs  and  yield  different  measures  of  effectiveness  (MOE).  The  first  step  in 
evaluating  these  alternatives  might  be  to  create  a  scatter  plot  of  the  effectiveness 
and  cost  of  the  different  alternatives  (A1  and  A2  in  Figure  7). 

EEOA:  “LEVEL  THE  PLAYING  FIELD” 

4.  Modified  Budget  Approach  (GOTO  1  &  3) 

Modify  alternatives  to  equalize  budget 
(Identify  vendor  MOE  responses  to  budget  increase) 
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In  the  absence  of  any  other  information — and  given  two  alternatives  A1  and 
A2  that  differ  in  both  costs  and  effectiveness,  one  strategy  for  selecting  a  vendor  is 
to  level  the  playing  field.  The  Modified-budget  Approach  asks  the  DM  to  determine 
the  highest-cost  alternative  they  are  willing  to  consider  (say  A2),  which  they  then  use 
as  a  notional  budget  for  the  program.  The  fourth  way  to  structure  an  EEoA 
recognizes  that  the  highest-cost  (highest-utility)  alternative  under  consideration  (A2 
in  Figure  7)  reveals  a  possible  budget  constraint. 

Then,  in  order  to  “level  the  playing  field,”  the  DM  can  ask  the  lower-cost, 
lower-utility  vendor  (that  offered  A1 )  how  they  might  use  the  extra  budget  available 
to  boost  the  utility  (MOE)  of  their  proposal.  Figure  7  indicates  the  response  from 
Vendor  1  is  that  with  the  same  budget  as  Vendor  2  they  are  able  to  boost  their  offer 
from  A1  to  Al*.'^®  Flaving  leveled  the  playing  field.  Vendor  1  becomes  the  clear 
winner  (i.e.,  since  for  the  same  budget,  offer  A1  *  has  a  higher  MOE  than  offer  A2). 
Note  that  this  approach  effectively  returns  the  problem  to  the  first  (and  third)  way  of 
structuring  an  EEoA:  maximizing  effectiveness  (MOE)  for  a  given  budget. 

5.  Modified-effectiveness  Approach 

Similarly,  the  fifth  way  to  structure  an  EEoA  levels  the  playing  field  for  a 
threshold  choice  of  utility  (MOE).  This  returns  the  problem  to  the  second  (and  third) 
way  of  structuring  an  EEoA;  minimizing  the  cost  of  achieving  a  given  level  of  MOE. 


Alternatively,  different  valuable  uses  for  the  money  saved  by  choosing  the  lower-cost  alternative 
could  be  brought  into  the  effectiveness  calculation.  Some  will  recognize  this  search  for  the  “next  best 
alternative  use  of  funds”  as  the  standard  economic  definition  of  opportunity  costs.  This  sets  the  stage 
for  the  sixth  way  to  structure  an  EEoA. 


-42- 


EOA:  “LEVEL  THE  PLAYING  FIELD” 


5.  Modified  Effectiveness  Approach  (GOTO  2  &  3) 
Modify  alternatives  to  equalize  MOE 
(Identify  vendor  COST  responses  to  higher  MOE  requirement) 


For  example,  Figure  8  shows  that  by  anchoring  the  desired  MOE  at  a  target 
level — such  as  that  offered  by  Vendor  2 — the  government  could  ask  Vendor  1  how 
much  it  would  cost  to  achieve  the  same  target  level  of  MOE.  In  Figure  8,  Vendor  1  is 
preferred,  since  the  response  (A1*)  minimizes  the  budget  required.'^’' 

6.  Opportunity-cost  (or  Effectiveness)  Approach 

Finally,  what  if  analysts  and/or  DMs  find  themselves  in  a  situation  in  which  i) 
alternatives  cannot  be  modified  to  obtain  response  functions,  and  ii)  future  funding  is 
unknown,  and  the  desired  level  of  MOE  cannot  be  determined?  In  this  case,  it  is 
likely  some  alternatives  cost  more  but  offer  more  effectiveness,  while  others  cost 
less  and  offer  less  effectiveness.  For  example,  see  Program  A  in  Figure  9. 


Note  that  the  response  function  for  Vendor  A  includes  points  A1  and  A1*. 
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6.  Opportunity  Cost  Approach 

(INTER-PROGRAM  Marginal  Analysis) 

A)  Question:  Where  is  the  extra  money  coming  from  if  I 
buy  the  high  cost  aiternative? 

B)  Question:  Where  is  the  extra  money  going  if  I  buy  the 
iow  cost  aiternative? 


MOE(Utils) 


Program  A 


MOE(Utils) 


Program  B 


A2 

o 


B2 

o 


A1 

o 


B1 

o 


Cost($) 


Cost($) 


Figure  9.  Opportunity-cost  Approach 

The  sixth  and  final  approach  to  structuring  an  EEoA  involves  an  inter-program 
comparison  called  the  Opportunity-cost  Approach.  Rather  than  modify  the 
alternatives  to  level  the  playing  field  as  in  the  EEoA  approaches  #4  and  #5,  the 
Opportunity-cost  Approach  requires  a  more  challenging  inter-program  analysis  to 
choose  between  lower-cost,  lower-effectiveness  alternatives  (such  as  A1  in  Figure 
9)  and  higher-cost,  higher-effectiveness  alternatives  (A2  in  Figure  9).  The  main 
challenge  in  selecting  an  alternative  in  this  context  is  that  the  DM  must  reach  beyond 
the  immediate  program  (A),  into  higher-level,  inter-program  considerations.  If 
alternatives  are  exogenously  determined,  and  it  is  not  possible  to  level  the  playing 
field,  then  to  find  the  most  cost-effective  solution  requires  information  about  other 
competing  programs  (i.e..  Program  B  in  Figure  9). 

This  involves  a  higher-level,  inter-program  analysis  similar  to  that  illustrated  in 
Figure  9.  The  DM  must  consider  the  loss  in  utility  (MOE)  in  other  programs  that 
might  be  sacrificed  (e.g.,  a  budget  cut  in  Program  B,  shifting  the  decision  from 
B2=>B1)  for  funds  to  be  released  to  purchase  greater  utility  (MOE)  in  the  program 
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under  review  (e.g.,  a  boost  in  the  budget  of  Program  A,  shifting  the  decision  from 
A1=>A2).  “[T]he  assessment  should  provide  details  as  to  how  excess  funding  [...] 
demands  will  be  accommodated  by  reductions  in  other  mission  areas,  or  in  other  [...] 
accounts”  (DoD,  2006,  July  7,  Section  3.2.2). 

Alternatively,  DMs  can  explore  how  much  more  utility  (MOE)  the  extra  money 
might  generate  somewhere  else  if  they  choose  the  low-cost  alternative  (A1  in 
Program  A).  These  are  tough  but  useful  questions  that  break  through  the  sub- 
optimization  of  most  traditional  AoAs.  As  a  consequence,  the  EEoA  encourages 
critical  communication  to  take  place  between  different  layers  of  the  organization  and 
a  seamless  interface  between  the  Requirements  Generation  System,  the  acquisition 
system,  and  PPBS.'^® 

The  bottom  line  is  that  it  is  often  more  transparent,  efficient,  and  effective  to 
develop  MOEs  that  are  independent  of  costs  and  to  treat  costs  as  an  independent 
variable  (CAIV).  Equally  important  are  the  roles  of  budget  (funding)  forecasts  and 
opportunity  costs  in  helping  structure  defense  investment  decisions.  By  structuring 
an  Economic  Evaluation  of  Alternatives  (EEoA)  using  one  of  the  six  approaches 
summarized  in  Figure  10,  decision-makers  can  help  achieve  the  primary  goal  of 
defense  acquisition  reform:  to  coordinate  Requirements  Generation,  Defense 
Acquisition,  and  PPBS  to  lower  costs  and  improve  performance  and  schedules. 


Fisher  (1965)  quotes  Secretary  of  Defense  Robert  McNamara:  “Suppose  we  have  two  tactical 
aircraft  which  are  identical  in  every  important  measure  of  performance  [MOE]  except  one — aircraft  A 
can  fly  ten  miles  per  hour  faster  than  Aircraft  B.  Thus,  if  we  need  about  1 ,000  aircraft,  the  total 
additional  cost  would  be  $10million.  If  we  approach  this  problem  from  the  viewpoint  of  a  given  amount 
of  resources,  the  additional  combat  effectiveness. .  .of  Aircraft  A  would  have  to  be  weighed  against  the 
additional  combat  effectiveness  which  the  same  $10million  could  produce  if  applied  to  other  defense 
purposes — more  Aircraft  B,  mor  or  better  aircraft  munitions,  or  more  ships,  or  even  more  military 
family  housing... This  kind  of  determination  is  the  heart  of  the  planning-programming- 
budgeting... problem  with  the  Defense  Department”  (p.  182). 
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VI.  Conclusion 


A.  A  Decision  Map  for  Decision-makers 

Given  future  public-spending  challenges,  the  affordability  of  defense 
investments  is  a  growing  concern.  As  a  consequence,  it  is  imperative  that  the  U.S. 
Department  of  Defense  (DoD)  obtain  the  best  value  for  every  dollar  it  invests  in  its 
Major  Defense  Acquisition  Programs  (MDAPs)  and  Major  Automated  Information 
Systems  (MAISs).  The  primary  goal  of  this  study  is  to  improve  defense  procurement 
decisions  by  bringing  the  taxpayer  up-front  alongside  the  warfighter  in  the  defense 
acquisition  process. 

This  study  identifies  several  major  challenges  that  face  senior  defense 
officials  as  they  attempt  to  select  the  best  vendor  for  an  MDAP  or  MAIS  through  a 
type  of  military  cost-benefit  analysis  called  an  “Analysis  of  Alternatives”  (AoA).  The 
study  reveals  a  significant  weakness  in  the  Multiple-criteria  Decision-making 
(MCDM)  approach  that  underpins  many  contemporary  AoAs.  The  weakness  is  that 
while  MCDM  techniques,  and  therefore  most  AoAs,  correctly  focus  on  lifecycle  costs 
and  the  operational  effectiveness  of  individual  alternatives,  affordability  is  often  an 
after-thought — only  implicitly  addressed  through  a  weight  assigned  to  costs. 

In  sharp  contrast,  the  Economic  Evaluation  of  Alternatives  (EEoA)  introduced 
in  this  study  offers  a  set  of  approaches  that  encourage  analysts  and  decision¬ 
makers  to  embed  affordability  assessments  directly  into  AoAs.  This  requires  working 
with  vendors  to  build  alternatives  based  on  different  funding  (budget/affordability) 
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scenarios.  Supported  by  a  static,  deterministic,  constrained-optimization,  micro- 
economic  (multi-stage  procurement  auction)  model,  this  EEoA  approach  explicitly 
addresses  affordability  up-front.'*® 

The  key  difference  between  the  MCDM  approach  to  AoAs  and  the  EEoA 
approach  is  that  instead  of  modeling  decision  alternatives  from  competing  vendors 
as  points  in  cost-effectiveness  space,  the  EEoA  models  alternatives  as  functions  of 
optimistic,  pessimistic  and  most-likely  funding  (resource  or  budget)  scenarios.  In 
demonstrating  how  to  embed  affordability  directly  into  an  AoA,  the  EEoA  represents 
an  important  step  in  the  long-running  effort  to  achieve  a  significant  defense 
acquisition  reform:  to  tightly  integrate  the  Requirements  Generation  System  and  the 
Defense  Acquisition  System  with  the  Planning,  Programming  and  Budgeting  System 
(PPBS). 

The  primary  goal  of  this  study  was  to  help  improve  public  investment 
decisions.  An  important  secondary  goal  was  to  develop  a  Decision  Map  to  help 
structure  the  Economic  Evaluation  of  Alternatives  (EEoA)  to  improve  defense 
acquisition  outcomes.  This  paper  provides  a  set  of  six  approaches  that  practitioners 
and  acquisition  officials  can  employ  to  structure  an  EEoA.  The  Decision  Map 
illustrated  in  Figure  10  below  offers  a  guide  for  senior  analysts  and  decision-makers 
to  select  which  of  the  six  EEoA  approaches  is  best  suited  to  their  circumstances. 


This  study  examines  two  popular  decision  criteria  used  by  the  military  to  structure  acquisition  decisions:  i) 
benefit/cost  ratios,  and  ii)  the  weighted  average  of  benefits  and  costs.  In  the  first  case,  the  study  cautions  that 
it  is  only  appropriate  to  use  benefit/cost  ratios  to  rank  alternatives  when  there  is  an  explicit  cost 
(funding/budget)  constraint  (or  effectiveness  constraint/threshold).  In  the  second  case,  this  study  advises 
defining  alternative  funding/budget  scenarios  in  advance  to  avoid  the  controversial  task  of  assigning  a  relative 
weight  to  the  cost/price  of  an  alternative.  Intuitively,  if  any  weight  whatsoever  is  assigned  to  cost,  then  this 
must  imply  that  there  is  an  overall  funding  constraint  (so  that  costs  matter),  or  equivalently,  that  there  is  some 
opportunity  cost  of  spending  money  to  pay  for  the  cost/price  of  an  alternative.  This  study  suggests  going  to 
the  source  of  the  problem,  either  by  identifying  the  likely  overall  budget/funding  constraint  for  the  military 
program,  or  by  identifying  likely  alternative  uses  of  the  funds.  This  approach  is  termed  the  Economic 
Evaluation  of  Alternatives  (EEoA). 
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In  conclusion,  this  study  offers  a  comprehensive  set  of  approaches  for 
procurement  officials  to  structure  public  investment  decisions.  Designed  to  improve 
acquisition  outcomes,  the  “Economic  Evaluation  of  Alternatives”  (EEoA)  addresses  a 
significant  weakness  in  most  contemporary  military  applications  of  the  current 
methodology — the  Analysis  of  Alternatives  (AoAs).  While  AoAs  correctly  focus  on 
lifecycle  costs  and  the  operational  effectiveness  of  alternatives,  “affordability”  is  an 
after-thought — at  best  only  implicitly  addressed  as  a  weight  placed  on  cost  in  the 
final  stages  of  the  analysis.  In  sharp  contrast,  the  EEoA  encourages  senior  analysts 
and  decision-makers  to  include  affordability  explicitly  and  up-front  in  structuring  an 
AoA.  This  requires  working  with  vendors  to  build  alternatives  based  on  a  reasonable 
spectrum  of  possible  funding  (budget  or  affordability)  scenarios.  The  Decision  Map 
below  provides  a  comprehensive  guide  to  EEoA  for  practitioners.  Finally,  this  study 
demonstrates  how,  by  embedding  affordability  directly  into  an  AoA,  the  EEoA 
approach  provides  a  unique  opportunity  for  senior  leadership  to  achieve  a  significant 
defense  acquisition  reform — to  integrate  Requirements  Generation  and  Defense 
Acquisition  with  PPBS,  to  reduce  future  costs  and  improve  performance  and 
schedules. 
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Figure  10.  Decision  Map  to  Structure  an  EEoA 
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